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Abstract 


Tata  Consultancy  Services  (TCS)  is  a  large  information  technology  consulting,  services,  and 
business  process  outsourcing  organization  interested  in  reducing  the  costs  of  conducting 
process  improvement  appraisals  at  its  multiple  locations.  TCS  initiated  a  pilot  to  determine 
whether  appraisals  could  be  performed  at  the  enterprise  level  instead  of  at  each  location  or 
center  while  preserving  the  integrity  of  the  Standard  CMMI®  Appraisal  Method  for  Process 
Improvement  V 1 . 1  (SCAMPIsm  V 1 . 1).  The  pilot  was  also  used  to  determine  whether  a  single 
type  of  appraisal  could  be  effective  in  an  organization  compliant  with  multiple  models. 

A  pilot  Class  A  SCAMPI  appraisal  for  the  People  Capability  Maturity  Model®  (People 
CMM®)  was  performed  jointly  with  a  Class  A  SCAMPI  appraisal  for  Capability  Maturity 
Model  Integration  (CMMI)  across  TCS  in  2004.  This  report  includes  the  draft  interpretation 
guide  used  for  four  mini-appraisal  pilots  and  the  final  enterprise-wide  Class  A  appraisal  at 
TCS.  The  information  in  this  report  could  serve  as  an  example  for  other  organizations  and  is 
fully  applicable  to  any  size  SCAMPI  appraisal  with  People  CMM. 


CMU/SEI-2005-SR-001 


vii 


VIII 


CMU/SEI-2005-SR-001 


1  Introduction 


Tata  Consultancy  Services  Limited  (TCS)  is  an  information  technology  consulting,  services, 
and  business  process  outsourcing  organization  based  in  Asia  but  with  locations  in  32 
countries  across  5  continents.  TCS,  like  many  large  organizations  with  multiple  software 
centers,  was  interested  in  reducing  the  costs  of  doing  process  improvement  appraisals  at  its 
multiple  locations  (18  separate  assessments  for  the  Software  CMM®  were  required  in  India 
alone)  and  across  multiple  models. 

TCS  has  also  had  its  own  Quality  Management  System  (QMS)  in  practice  since  1989.  This 
system,  called  iQMS,  was  written  to  be  model  independent  and  compliant  with  ISO  9000,  the 
Software  CMM  (SW-CMM),  Capability  Maturity  Model®  Integration  (CMMI®),  People 
CMM,  and  the  Malcolm  Baldrige  National  Quality  Award  (MBNQA).  TCS  developed  this 
system  to  maintain  a  high  quality  focus  without  being  pulled  by  changing  quality  models. 
Rather,  as  new  models  are  developed  and  existing  models  evolve,  iQMS  can  evolve  in 
concept  and  not  be  limited  by  model  specifics  and  terminology. 

1.1  Defining  a  Pilot 

The  organization  approached  Ron  Radice  to  discuss  whether 

•  appraisals  could  be  performed  at  the  enterprise  level  instead  of  at  each  location  or  center 
while  still  preserving  the  integrity  of  the  Standard  CMMI  Appraisal  Method  for  Process 
Improvement  (SCAMPIsm)  appraisal  method 

•  a  single  type  of  appraisal  could  be  effective  in  an  organization  that  was  compliant  with 
multiple  models  but  strategically  looking  to  become  independent  of  external  quality 
models 

Radice  believed  it  was  possible  and  began  working  with  TCS  on  a  pilot  to  explore  the  use  of 
one  common  appraisal  method  across  multiple  quality  models,  including  the  SW-CMM, 
CMMI,  and  the  People  CMM.  Their  approach,  which  they  called  an  enterprise  appraisal,  also 
needed  to  be  fully  consistent  with  a  Class  A  SCAMPI  appraisal. 

1 .2  Using  SCAMPI  with  the  People  CMM 

When  the  pilot  began,  TCS  was  considering  the  move  from  SW-CMM  to  CMMI,  but  was 
concerned  about  the  cost  of  migrating  at  18  locations.  TCS  had  completed  four  People  CMM 
V2.0  appraisals  and  was  thinking  of  moving  all  its  locations  to  People  CMM  V2.0.  A  natural 


CMU/SEI-2005-SR-001 


1 


starting  point  for  the  project  was  investigating  the  use  of  one  appraisal  method  with  both 
CMMI  and  People  CMM. 

Because  the  CMMI  Product  Team  had  invested  a  considerable  amount  of  time  and  energy  in 
CMMI  and  the  related  SCAMPI  appraisal  method,  it  seemed  natural  to  try  to  determine 
whether  SCAMPI  could  be  applied  to  other  quality  models.  Radice,  who  had  previously 
performed  SCAMPI  with  SW-CMM,  knew  SCAMPI  was  rich  enough  to  be  used  with  more 
than  one  model  [Radice  03].  The  CMMI  Steering  Group  also  has  sanctioned  SCAMPI’s 
application  with  SW-CMM,  confirming  its  usefulness  with  models  other  than  CMMI.  Radice 
decided  to  perform  a  pilot  Class  A  appraisal  to  meet  TCS’s  objectives  and  to  provide 
evidence  that  SCAMPI  could  work  for  a  People  CMM  appraisal. 

Because  the  SW-CMM  was  one  of  the  main  sources  for  CMMI,  the  models  have  much  in 
common.  People  CMM,  on  the  other  hand,  is  very  different  from  CMMI.  Even  though  the 
architecture  is  the  same  across  these  models  (e.g.,  they  both  use  maturity  levels,  process 
areas,  goals,  and  related  practices),  the  focus  is  different.  CMMI  concentrates  on  software 
and  systems,  while  the  People  CMM  is  focused  on  the  workforce.  However,  even  though 
CMMI  does  not  directly  integrate  the  People  CMM,  there  are  similarities  and  synergies 
between  the  process  areas  and  practices  of  the  two  models  [Hefley  03]. 

A  further  challenge  arose  because  different  terms  are  used  for  common  ideas  in  CMMI  and 
the  People  CMM,  and  in  some  cases  the  same  terminology  is  used  for  different  concepts. 
Further,  there  are  different  views  about  how  objective  evidence  should  be  obtained  and  used 
(e.g.,  SCAMPI  using  CMMI  requires  direct  and  indirect  artifacts  which  are  derivable  from 
the  CMMI  practice  descriptions  and  the  interpretive  material  in  the  model,  but  these  are  not 
so  clearly  derived  from  People  CMM).  The  People  CMM  appraisal  method  made  strong  use 
of  surveys  as  primary  evidence.  While  SCAMPI  permits  the  use  of  surveys  and  other  similar 
instruments,  it  is  careful  to  raise  the  risk  that  surveys  should  not  be  used  to  replace 
affirmations  through  interviews. 

The  team  chose  to  work  within  these  challenges  under  the  guiding  premise  that  to  be 
SCAMPI  CMMI  compliant,  no  requirements  of  SCAMPI  could  or  would  be  dropped  when 
using  SCAMPI  with  People  CMM.  Because  SCAMPI  did  not  address  all  of  the  People 
CMM’s  needs  (e.g.,  a  review  of  the  draft  with  the  organization’s  legal  department),  CMMI 
would  need  to  be  extended  to  address  People  CMM  appraisal  needs.  Again  no  extension  to 
the  People  CMM  baseline  would  compromise  any  SCAMPI  requirement. 

Because  the  methods  are  the  intellectual  property  of  Carnegie  Mellon  University,  agreement 
was  needed  to  determine  if  the  application  of  SCAMPI  using  People  CMM  would  be 
accepted  in  concept.  Once  Bill  Peterson,  Director  of  the  Software  Engineering  Process 
Management  Program  at  the  SEI,  gave  permission  to  proceed  with  a  pilot  using  SCAMPI 
with  People  CMM,  Radice  began  working  on  interpretation  guidelines  for  the  pilot. 
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2  Guidelines  for  the  Appraisal 


A  draft  interpretation  guide  was  developed  to  show  how  the  SCAMPI  method  could  be 
applied  with  the  People  CMM,  v  2.0  [Curtis  02]  as  the  reference  model  for  a  process 
appraisal.  The  guide  was  developed  using  material  from  the  Standard  CM  MI  Appraisal 
Method  for  Process  Improvement  (SCAMPI),  Version  1.1:  Method  Definition  Document  [SEI 
Ola]  and  the  People  CMM-Based  Assessment  Method  Description  [Hefley  98]. 

To  keep  the  work  simple,  the  SCAMPI  Method  Definition  Document  (MDD)  was  used  as  the 
basis  for  the  SCAMPI  with  People  CMM  interpretation  guide  for  the  People  CMM  such  that: 

1.  When  the  SCAMPI  MDD  was  applied  fully  “as  is”  for  People  CMM,  the  wording  was 
not  changed,  but  kept  as  in  SCAMPI. 

2.  When  People  CMM  used  different  terminology,  the  terms  were  changed  in  the 
interpretation  guide  as  indicated  in  Appendix  L. 

3.  In  some  cases,  material  has  been  added  from  the  People  CMM-Based  Assessment 
Method  Description  (People  CMM  BAM)  [Hefley  98]  to  provide  a  clearer 
understanding  for  SCAMPI  with  People  CMM.  Comparisons  with  People  CMM  BAM 
are  shown  in  Appendix  D. 

4.  The  Preparing  phase  in  People  CMM-BAM  Description  is  the  same  temporal  phase  as 
the  SCAMPI  Plan  and  Prepare  for  Appraisal  phase  in  the  SCAMPI  MDD.  The  People 
CMM-BAM  Description  has  7  tasks  in  this  phase,  while  SCAMPI  has  5  processes  with 
21  activities. 

The  sponsor’s  objectives  for  performing  SCAMPI  were  determined  in  the  first  process  of  the 
first  phase.  All  other  planning,  preparation,  execution,  and  reporting  of  results  proceeded 
from  this  initial  activity  according  to  the  phase  and  processes  outlined. 

The  following  sections  in  this  chapter  present  the  process  definitions  used  to  perform  the 
pilot  enterprise  appraisal. 
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2.1  Phase  1 

This  phase  has  five  processes:  Analyze  Requirements,  Develop  Appraisal  Plan,  Select  and 
Prepare  Team,  Obtain  and  Analyze  Objective  Evidence,  and  Prepare  for  Collection  of 
Objective  Evidence.1 

2.1.1  Analyze  Requirements 

Purpose— To  understand  the  business  needs  of  the  organization  for  which  the  appraisal  is 
being  requested.  The  appraisal  team  leader  should  collect  information  and  help  the  appraisal 
sponsor  match  appraisal  objectives  with  their  business  objectives. 

Entry  Criteria 

•  An  appraisal  sponsor  has  decided  that  a  SCAMPI  with  People  CMM  appraisal  should  be 
performed. 

•  People  who  can  provide  statements  of  requirements  for  the  appraisal  are  available. 

Inputs 

•  sponsor 

•  initial  requirements  and  constraints 

•  process-related  legacy  information 

The  use  of  legacy  information  from  prior  appraisals  and  mini-appraisals  within  a  reasonable 
time  period  is  allowed.  This  is  new  for  SCAMPI  with  People  CMM.  Legacy  information 
can  be  an  aid  to  formal  appraisals  during  the  discovery  activities  (i.e.,  document  reviews). 

Thus  when  information  from  a  previous  document  review  retains  evidence  that  is  reusable 
during  a  future  appraisal,  it  should  be  considered. 

However,  not  all  documents  will  become  legacy  information  for  reuse.  The  appraisal  team 
must  always  ask  to  make  sure  the  information  is  current:  if  a  policy  or  procedure  has  not 
changed  between  two  appraisals  or  mini-appraisals,  the  process-related  legacy  information 
can  be  reused.  What  typically  cannot  be  reused  is  information  demonstrating  process 
performance.  For  example,  a  development  plan  for  an  individual  might  have  changed,  or  a 
project  plan  might  have  been  revised  or  a  project  closed  out.  In  cases  like  these,  new 
documentation  is  required. 

Activities — The  five  activities  listed  below  are  required  for  this  process. 

2. 1 . 1 . 1  Determine  Appraisal  Objectives 

2.1. 1.2  Determine  Appraisal  Constraints 

2.1. 1.3  Determine  Appraisal  Scope 

1  The  SCAMPI  CMMI  processes  (such  as  Analyze  Requirements)  are  not  at  an  equivalent  level  of  comparison 
with  the  People  CMM-BAM  description.  Comparisons  are  made  at  the  activity  level  in  SCAMPI  to 
maintain  parity  across  the  two  methods. 
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2. 1 . 1.4  Determine  Outputs 

2. 1.1.5  Obtain  Commitment  to  Appraisal  Input 

Outputs — The  appraisal  input. 

Outcome — The  decision  to  proceed  with  the  appraisal  based  on  a  shared  understanding  of 
the  appraisal  objectives,  constraints,  outputs,  and  scope. 

Exit  Criteria 

•  Initial  contact  between  the  appraisal  sponsor  and  authorized  SCAMPI  with  People  CMM 
Lead  Appraiser  has  occurred. 

•  The  Lead  Appraiser  has  been  given  access  to  members  of  the  sponsoring  organization. 

•  The  appraisal  input  has  been  approved  by  the  appraisal  sponsor  and  placed  under  change 
management. 

Key  Points— At  this  early  stage  in  the  process,  gathering  information  that  supports  good 
planning  is  most  important.  Often,  the  appraisal  team  leader  must  educate  members  of  the 
sponsor’s  organization  about  the  purpose  and  role  of  appraisals. 

Tools  and  Techniques — Collaborative  consultation  between  the  appraisal  team  leader  and 
the  appraisal  sponsor  is  important  in  this  activity.  The  appraisal  team  leader  might  need  to 
interview  the  sponsor  to  get  the  needed  information  and  reach  agreements.  In  some  settings,  a 
series  of  meetings  with  different  stakeholders  might  be  needed  to  elicit  and  build  consensus 
on  the  business  needs  that  can  be  met  through  a  SCAMPI  with  People  CMM  appraisal. 

Understanding  the  history  of  appraisals  in  the  organization— especially  the  organizational  and 
model  scope  of  past  appraisals — is  important  in  understanding  the  requirements  for  the 
appraisal  under  consideration.  The  choices  sponsors  make  about  appraisal  scope  are  often 
tied  to  their  priorities  for  process  improvement,  which  are  sometimes  unstated. 

A  history  reflecting  all  appraisal  activities  is  captured  in  the  SCAMPI  appraisal  plan  using  the 
SCAMPI  with  People  CMM  template,  which  is  similar  to  the  template  used  for  SCAMPI. 
This  history  sets  a  foundation  for  sampling  the  organization  to  ensure  sufficient 
representation  during  an  appraisal.  Refer  to  Appendix  A  for  more  information  about 
sampling. 

Metrics— The  following  metrics  support  the  appraisal  team  leader’s  monitoring  of  this  work: 

•  calendar  time  between  initial  contact  and  finalization  of  requirements 

•  effort  expended  to  gather  and  analyze  requirements 

•  number  of  meetings  with  representatives  of  the  sponsoring  and/or  appraised  organization 
Metrics  are  captured  in  the  appraisal  plan. 
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Verification  and  Validation— The  exit  criterion  for  this  activity  is  the  formal  approval  of  the 
appraisal  input  and  its  placement  under  change  management.  Appraisal  input  is  captured 
using  a  template  similar  to  the  SCAMPI  CMMI  template  and  migrated  to  the  appraisal  plan. 

Review  of  the  documented  agreements  resulting  from  this  set  of  activities  validates  the 
requirements,  which  feed  into  appraisal  planning. 

Records — The  appraisal  input. 

Tailoring — The  experience  of  the  sponsor  with  process  appraisals  drives  tailoring  choices  for 
this  process.  A  relatively  inexperienced  appraisal  sponsor  needs  a  great  deal  of  information 
and  consultation  to  provide  meaningful  and  complete  requirements  for  the  appraisal. 

Experienced  sponsors  might  have  overly  aggressive  requirements. 

Risks  determined  during  this  process  are  captured  in  the  SCAMPI  with  People  CMM 
appraisal  plan. 

Interfaces  with  Other  Processes — This  process  is  a  foundation  for  the  success  or  failure  of 
the  entire  appraisal.  At  this  point  in  the  appraisal,  the  greatest  opportunity  for  avoiding 
problems  downstream  exists.  Gathering  and  understanding  the  requirements  for  conducting  a 
SCAMPI  with  People  CMM  appraisal  is  vital  to  making  appropriate  decisions  and  providing 
value  to  the  sponsor.  Many  examples  of  problems  encountered  during  appraisals  can  be 
traced  to  shortcomings  in  the  conduct  of  this  process.  The  extent  to  which  the  activities 
described  here  are  distinct  from  the  activities  described  in  process  2.1.2  Develop  Appraisal 
Plan  will  depend  on  the  strategy  and  preferences  of  both  the  appraisal  team  leader  and  the 
appraisal  sponsor. 

Summary  of  Activities — The  objectives  that  motivate  the  conduct  of  an  appraisal  must  be  well 
understood  so  appropriate  participants,  tailoring  decisions,  and  appraisal  outputs  can  be  selected.  The 
constraints  that  shape  the  appraisal  enactment,  in  light  of  the  objectives,  might  limit  achievement  of 
the  desired  result  if  they  are  not  adequately  understood  and  negotiated.  A  clear  agreement  regarding 
appraisal  outputs  and  their  intended  use  will  help  maintain  the  sponsorship  needed  for  conducting  the 
appraisal  and  acting  on  the  results.  Agreement  on  these  objectives,  constraints,  outputs,  and  intended 
use  forms  the  basis  for  a  commitment  to  the  plan  for  conducting  the  appraisal. 

2.1.1 .1  Determine  Appraisal  Objectives 

Activity  Description —  When  using  People  CMM,  the  business  needs  for  process  improvement  drive 
the  requirements  for  the  conduct  of  the  appraisal  and  generally  include  one  or  more  of  the  following 
closely  related  factors: 

•  establishment  of  a  baseline  characterization  of  workforce  practices  to  enable 
improvement 

'•  organizational  merge,  transition,  or  change  in  ownership 

•  sustaining  higher  CMM  or  CMMI  levels  through  solid  People  CMM  workforce  practices 
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The  fundamental  premise  of  process  improvement  is  that  organizational  processes 
significantly  impact  these  factors.  A  fair  and  objective  characterization  of  the  processes  in  use 
in  the  organization  is  the  essential  reason  for  conducting  an  appraisal.  In  addition  to  this 
motivation,  a  sponsor’s  desire  to  conduct  an  appraisal  could  be  driven  by  one  or  more  of  the 
following  business-related  objectives: 

•  to  document  a  credible  benchmark  that  reflects  successful  process  improvement 

•  to  evaluate  areas  of  potential  risk  that  might  affect  the  performance  of  the  organization 

•  to  involve  members  of  the  appraised  organization  in  improving  the  performance  of  the 
process 

•  to  support  specific  decisions  related  to  the  direction  of  a  new  or  existing  improvement 
program 

Required  Practices 

•  Identify  sponsor  and  relevant  stakeholders  and  establish  communication. 

•  Document  business  and  appraisal  objectives. 

•  Ensure  the  alignment  of  appraisal  objectives  to  business  objectives. 

•  Determine  and  document  the  appraisal  usage  mode  as  “internal  process  improvement.” 

•  Make  organization  aware  of  the  forthcoming  appraisal.  Achieving  early  buy-in  from 
members  in  the  organization  is  easier  if  they  are  aware  of  the  forthcoming  appraisal. 

The  results  of  these  practices  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — At  least  one  communication  between  the  appraisal  team  leader  and 
sponsor  is  required. 

Optional  Practices — None. 

Implementation  Guidance — Organizations  with  experience  in  the  use  of  appraisals  might 
have  a  clear  set  of  appraisal  objectives  identified  before  contacting  a  Lead  Appraiser.  While 
this  provides  the  Lead  Appraiser  with  a  starting  point,  it  does  not  permit  him  or  her  to  skip 
this  activity. 

Also  note  that  the  appraisal  sponsor  and  senior  site  manager  might  be  the  same  person. 
Depending  on  the  structure  of  the  appraised  organization  and  the  usage  mode,  it  is  often 
important  to  distinguish  the  role  of  senior  site  manager  from  that  of  appraisal  sponsor.  For 
some  appraisals,  these  two  roles  are  encompassed  in  the  duties  of  a  single  person.  For  other 
appraisals,  these  two  roles  might  be  held  by  two  people  working  many  time  zones  away  from 
each  other. 

2.1.1 .2  Determine  Appraisal  Constraints 

Activity  Description — The  constraints  under  which  the  appraisal  must  be  conducted  are 
determined  based  on  a  dialog  between  the  appraisal  team  leader  and  the  appraisal  sponsor 
and/or  senior  site  manager.  This  is  typically  an  iterative  process  in  which  the  preferences  of 
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the  appraisal  sponsor,  the  limits  of  the  method,  and  the  consequent  resource  requirements  are 
balanced  against  each  other  to  arrive  at  an  optimal  set  of  appraisal  input  parameters. 

Required  Practices 

•  Establish  high-level  cost  and  schedule  constraints. 

•  Determine  which  process  areas  (PAs)  and  organizational  entities  are  to  be  included. 

•  Determine  the  minimum,  maximum,  or  specific  sample  size  or  coverage  that  is  desired 
for  the  appraisal. 

•  Negotiate  constraints  and  objectives  with  stakeholders  to  ensure  feasibility. 

•  Document  negotiated  constraints  to  be  met. 

•  Plan  coordination  for  integration  of  SCAMPI  with  People  CMM  with  other  SCAMPI 
methods  (e.g.,  SCAMPI  for  CMMI),  as  appropriate.  See  Appendix  I  for  a  discussion  of 
these  coordination  factors. 

•  Obtain  a  People  CMM  appraisal  kit. 

•  Complete  the  organizational  characteristics  questionnaire. 

Cost  and  schedule  constraints  are  captured  in  the  appraisal  plan.  Constraints  and  objectives 
will  also  be  captured  in  the  appraisal  plan,  including  those  to  be  met,  managed,  or  accepted  as 
is. 

Parameters  and  Limits — At  least  one  communication  between  the  appraisal  team  leader  and 
the  sponsor  is  required.  Cost  and  schedule  constraints  identified  during  this  early  stage  of  the 
appraisal  are  expected  to  be  high-level — not  detailed — estimates.  They  might  take  the  form 
of  statements  such  as  “We  need  this  done  in  Q4,”  “You  can’t  use  more  than  five  of  my  people 
on  the  team,”  or  “I  can’t  afford  to  have  it  last  more  than  a  month.”  Constraints  identified  by 
the  appraisal  input  must  be  negotiated  between  the  sponsor  and  the  appraisal  team  leader. 

Optional  Practices — Document  the  rationale  for  choices  made  and  the  associated  tradeoffs 
as  a  resource  for  later  planning  and  future  appraisals.  Choices  and  associated  tradeoffs  will  be 
captured  in  the  appraisal  plan. 

Implementation  Guidance — Practical  limitations  relating  to  time,  cost,  and  effort  are 
clarified  and  negotiated  in  the  context  of  other  requirements  the  sponsor  has.  The  business 
context  in  which  the  appraisal  is  conducted  drives  the  choices  of  the  appraisal  team  leader. 
Appraisals  should  not  be  conducted  in  isolation  from  other  activities  relating  to  process 
management  and  improvement.  The  needs  of  related  stakeholders  often  place  requirements 
on  the  conduct  of  the  appraisal. 

Obtain  a  People  CMM  appraisal  kit  early  for  training  and  preparing  the  appraisal  team. 
Answers  given  on  the  organizational  characteristics  questionnaire  can  inform  help  to  identify 
constraints  that  will  define  the  scope  of  the  appraisal. 
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If  the  decision  is  to  integrate  SCAMPI  with  People  CMM  with  other  SCAMPI  methods,  this 
should  be  reviewed  and  analyzed  as  early  as  possible  since  it  could  affect  the  performance  of 
other  method  activities  in  SCAMPI  with  People  CMM. 

2.1. 1.3  Determine  Appraisal  Scope 

Activity  Description — The  appraisal  scope  consists  of  the  reference  model  scope  and  the 
organizational  scope.  The  reference  model  scope  must  be  determined  and  documented  early 
in  the  planning  process.  The  appraisal  team  leader  is  responsible  for  ensuring  that  the  sponsor 
makes  an  informed  choice  regarding  the  PAs  included  in  the  scope  of  the  appraisal.  The 
selection  of  appraisal  outputs  should  be  driven  by  the  understanding  of  their  intended  use 
established  during  the  requirements  analysis  activity  and  might  dictate  some  selections  in 
model  scope.  The  organizational  scope  defines  the  bounds  of  the  organization  to  be 
investigated  during  the  appraisal. 

Instantiations  are  selected  as  representative  of  the  organization  and  the  contexts  in  which 
processes  are  implemented.  Reconciling  the  interactions  between  model  scope  and 
organizational  scope  is  an  important  part  of  this  activity  (i.e.,  for  practices  implemented  by 
units  and  related  workgroups,  each  unit  and  related  workgroups;  for  practices  implemented 
organization-wide,  the  instance).  A  particular  organizational  scope  begets  a  particular  model 
scope;  a  particular  model  scope  requires  a  particular  organizational  scope. 

Note  that  there  is  a  key  distinction  between  the  relevance  of  projects  in  the  SCAMPI  model 
and  the  People  CMM.  The  equivalent  of  “projects”  in  People  CMM  is  organizational  units. 
Thus,  when  using  SCAMPI  with  People  CMM,  the  unit  and  related  workgroups  must  be 
included  in  the  appraisal  process  scope. 

“Unit”  has  recursive  meanings  as  defined  in  People  CMM,  so  “unit”  can  be  any 
organizational  entity  including  and  above  a  project  in  some  organizations.  See  Appendix  E 
for  more  information.  Units  are  subsets  of  an  organization,  and  projects  and/or  workgroups 
exist  within  them.  Here,  “workgroup”  is  used  the  same  way  it  is  used  in  the  People  CMM 
(i.e.,  workgroup  or  work-group  like).  In  SCAMPI  with  People  CMM,  “project”  is  equivalent 
to  “unit”  in  People  CMM. 

Required  Practices 

•  Determine  and  document  the  reference  model  scope  to  be  used  for  the  appraisal. 

•  Determine  and  document  the  appraised  organizations  and  entities  to  be  investigated 
during  the  appraisal. 

•  Select  the  appraisal  implementation  method: 

-  People  CMM  survey  only,  which  is  a  Class  B  appraisal  and  not  addressed  in  this 
version  of  the  guide 

-  stand-alone  SCAMPI  with  People  CMM  appraisal,  which  is  a  Class  A  appraisal 

-  SCAMPI  with  People  CMM  appraisal  integrated  with  SCAMPI  with  CMMI  or  other 
assessment/appraisal  methods,  which  is  also  a  Class  A  appraisal 
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•  If  the  appraisal  plan  was  not  started  in  the  previous  activity  it  should  be  done  at  this  time. 

Parameters  and  Limits — The  reference  model  scope  includes  the  PAs  and  associated 
maximum  maturity  level  investigated  by  the  appraisal  team  (i.e.,  the  goals  that  will  be  rated 
for  each  PA  within  the  scope  of  the  appraisal). 

The  model  scope  of  the  appraisal  must  encompass  at  least  one  PA.  All  goals  up  to  and 
including  the  maturity  level  for  each  selected  PA  must  be  rated;  individual  goals  within  a  PA 
cannot  be  excluded.  The  appraisal  implementation  method  should  be  identified. 

Instantiations  must  be  representative  of  the  implemented  processes  and  functional  areas  being 
investigated  within  the  appraised  organization  and  operate  within  a  coherent  process  context 
(see  glossary  for  the  full  definition  of  “instantiation”).  This  is  also  sometimes  known  as  the 
organizational  scope  of  the  appraisal.  The  rationale  for  selecting  these  elements  as 
representative  of  the  appraised  organization  should  be  documented. 

Typically,  the  appraised  organization  will  be  specified  so  that 

•  at  least  two  instances  of  the  processes  being  investigated  are  available  as  sources  of 
objective  evidence 

•  a  representative  coverage  of  the  units  active  within  the  organization  is  obtained 

Selection  of  instantiations  within  the  appraised  organization  might  be  accomplished  through  a 
survey  (when  chosen  as  an  instrument  for  an  appraisal)  or  through  summarizing  information 
learned  from  discussions  with  members  of  the  organization.  For  processes  enacted  at  the 
organization  level  (such  as  workforce  planning),  multiple  instances  are  not  required. 

The  representative  instantiations  to  be  investigated  during  the  appraisal  will  also  drive  the 
selection  of  participants  needed  to  provide  sources  of  objective  evidence.  An  initial 
determination  of  appraisal  participants,  by  name  and  role,  should  be  negotiated  with  the 
appraisal  sponsor  and/or  the  senior  site  manager  as  part  of  the  early  determination  of 
organizational  scope.  This  is  refined  later  during  detailed  appraisal  planning. 

All  of  these  initially  determined  participants  are  only  potential  participants.  The  actual 
participants  are  determined  prior  to  the  appraisal  and  as  late  as  possible  to  maintain  more 
randomness  in  representative  selection.  Some  participants,  such  as  process  owners  and  some 
managers,  need  to  be  identified  earlier  due  to  their  roles  in  the  organization,  and  it  is  not 
likely  these  roles  will  change. 

Optional  Practices — Use  broad-based  survey  instruments  or  a  practice  implementation 
indicator  (PII)  database  to  characterize  the  population  of  units  and  workgroups  in  an 
organization  before  determining  the  organizational  scope  of  the  appraisal. 

Implementation  Guidance — There  are  two  primary  parameters  of  the  appraisal  enactment 
that  contribute  significantly  to  the  resulting  cost  (in  terms  of  effort):  the  PA  scope  of  the 
People  CMM  model  encompassed  by  the  appraisal  and  the  number  and  size  of  units  and 
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workgroups  selected.  While  other  parameters  contribute  to  the  cost  and  schedule  of  an 
appraisal,  these  two  scope  parameters  provide  the  greatest  opportunity  to  shape  the 
magnitude  of  the  appraisal. 

SCAMPI  with  People  CMM  requires  that  findings  for  the  appraised  organization  be  derived 
from  objective  evidence  on  the  implementation  of  practices  collected  from  each  of  the 
organizational  process  instantiations  included  in  the  appraisal.  The  size  and  number  of 
instantiations  investigated  are  selected  to  form  a  valid  sample  of  the  appraised  organization  to 
which  the  results  are  attributed. 

Clearly,  a  broadly  defined  appraised  organization  (e.g.,  a  multinational  enterprise)  requires 
collecting  and  analyzing  significantly  more  objective  evidence  than  a  narrowly  defined 
appraised  organization  (e.g.,  a  specific  product  line  within  a  specific  business  unit  at  a  single 
geographical  location). 

The  organization  to  which  appraisal  results  are  attributed  should  be  described  accurately  in 
all  statements  made  by  the  appraisal  team  leader  and  sponsor.  It  is  the  responsibility  of  the 
appraisal  team  leader  to  understand  the  larger  organizational  context  in  which  the  appraised 
organization  resides.  The  appraised  organization  might  be  a  subset  of  a  larger  organization 
(e.g.,  a  division  or  location  within  a  company).  Familiarity  with  the  nature  of  departmental 
structures,  matrixed  subject  matter  expert  groups,  integrated  product  teams,  program  or 
project  and  unit  groupings,  or  product  line  implications  that  might  affect  the  interpretation  of 
appraisal  outcomes  will  aid  in  obtaining  this  understanding  when  selecting  units  and 
workgroups. 

The  selection  of  units  and  workgroups  might  not  be  required  in  some  organizations. 

However,  when  an  organization  is  large,  sampling  might  be  necessary  to  make  the  appraisal 
representative  and  practical  to  perform. 

The  appraisal  team  leader  should  work  with  representatives  from  the  organization  to 
document  a  clear  statement  of  the  model  and  organizational  scope  of  the  appraisal.  The 
model  scope  should  be  documented  using  a  list  of  PAs  to  be  included  in  the  appraisal,  as  well 
as  the  model  components  to  be  rated  by  the  appraisal  team.  The  organizational  scope  of  the 
appraisal  should  be  documented  in  the  clearest  terms  possible,  given  the  nature  of  the 
organizational  structure  in  place.  It  is  often  difficult  to  specify  unambiguous  boundaries 
without  naming  individuals  in  some  organizations.  Information  about  the  appraised 
organization  should  be  documented  in  a  way  that  allows  future  appraisal  sponsors  to  replicate 
exactly  the  scope  of  the  organization  appraised.  This  information  should  be  in  the  appraisal 
plan  and  used  in  briefing  the  appraisal  team  and  appraisal  participants  (in  summary  form  if 
needed). 
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2.1 .1.4  Determine  Outputs 

Activity  Description — Identify  the  specific  appraisal  outputs  to  be  produced.  Some  appraisal 
outputs  are  required  and  additional  outputs  are  tailorable  (see  Parameters  and  Limits  and 
Optional  Practices  in  this  activity). 

Obtain  clear  answers  to  the  following  questions: 

•  What  ratings  will  be  generated  during  the  appraisal? 

•  Will  a  final  report  be  written  to  document  appraisal  results? 

•  Will  recommendations  for  addressing  specific  findings  be  generated  and  reported? 

Required  Practices 

•  Review  required  SCAMPI  with  People  CMM  outputs  with  the  appraisal  sponsor.  Review 
and  select  optional  SCAMPI  with  People  CMM  outputs  with  the  appraisal  sponsor. 

•  Determine  the  recipients  of  appraisal  outputs. 

Parameters  and  Limits— Required  SCAMPI  with  People  CMM  outputs  include  the 
following: 

•  appraisal  record  (see  activity  2.3.2.2  Generate  Appraisal  Record) 

•  appraisal  disclosure  statement  (see  activity  2.2.4.4  Document  Appraisal  Results) 

•  People  CMM  Steward  Data  (see  activity  2.3.2.3  Provide  Appraisal  Feedback  to  People 
CMM  Steward) 

As  stated  in  the  Assessment  Requirements  for  CMMI  (ARC),  all  the  goals  for  the  process 
area  or  areas  investigated  by  the  team  must  be  rated,  although  the  choice  might  be  made  to 
not  disclose  the  ratings  to  anyone  other  than  the  appraisal  sponsor.  At  a  minimum,  the 
sponsor  gets  the  following  appraisal  outputs: 

•  final  findings,  including  statements  of  strengths  and  weaknesses  documented  by  the  team 
for  every  PA  investigated 

•  all  ratings  planned  for  and  generated  by  the  team. 

Decisions  reached  on  appraisal  outputs,  including  what  ratings  are  to  be  reported,  are 
documented  in  the  appraisal  input. 

Optional  Practices — The  appraisal  sponsor  might  request  that  additional  rating  outputs  be 
generated  as  a  result  of  the  appraisal.  Typical  rating  outputs  that  might  be  selected  include  the 
following: 

•  maturity  level  ratings 
.  •  practice  ratings 

•  an  option  to  use  “partially  satisfied”  as  a  rating  assigned  to  a  PA 

•  other  (non-typical)  outputs 

To  —  ~  CMU/SEI-2005-SR-001 


Many  of  these  optional  appraisal  outputs  are  discussed  further  in  process  2.2.4  Generate 
Appraisal  Results. 

The  sponsor  might  also  request  that  other  products  be  generated  as  appraisal  outputs.  Typical 
products  that  might  be  requested  include  the  following  (see  activity  2.3. 1.3  Plan  for  Next 
Steps): 

•  appraisal  final  report 

•  recommendations  for  taking  action  on  the  appraisal  results 

•  process  improvement  action  plan 

Implementation  Guidance — Goal  satisfaction  ratings  of  the  PAs  within  the  scope  of  the 
appraisal  are  a  minimum  requirement.  Maturity  level  ratings  are  optional.  Reporting  the 
ratings  to  the  appraisal  participants  is  not  required  even  though  ratings  are  performed.  The 
sponsor  has  sole  authority  to  decide  (in  advance)  which  ratings  will  or  will  not  be  reported, 
and  to  whom  they  will  be  reported. 

While  statements  of  findings  are  a  required  output  of  the  method,  creating  a  written  report 
that  elaborates  on  the  findings  is  optional.  The  sponsor  should  decide  if  resources  are  to  be 
spent  creating  this  artifact.  Similarly,  the  task  of  creating  recommendations  to  address  issues 
uncovered  in  the  appraisal  might  require  expertise  that  is  not  represented  on  the  appraisal 
team.  The  characteristics  of  the  appraised  organization  and  the  constraints  that  shape  its 
improvement  program  should  be  carefully  considered  when  making  process  improvement 
recommendations. 

2.1 .1.5  Obtain  Commitment  to  Appraisal  Input 

Activity  Description — The  appraisal  sponsor  formally  approves  the  appraisal  input,  and  this 
set  of  information  is  placed  under  change  management. 

Required  Practices 

•  Record  required  information  in  the  appraisal  input  record. 

•  Obtain  sponsor  approval  of  the  appraisal  input  record  and  appraisal  plan  as  presently 
defined. 

•  Manage  changes  to  the  appraisal  input,  obtaining  sponsor  approval  of  changes. 

Parameters  and  Limits— The  appraisal  input  might  be  generated  incrementally  throughout 
planning,  but  must  be  approved  prior  to  the  start  of  data  collection.  At  a  minimum,  the 
appraisal  input  should  provide  the  information  needed  to  address  the  following: 

•  the  identity  of  the  appraisal  sponsor  and  the  relationship  of  the  sponsor  to  the  appraised 
organization 

•  the  appraisal  purpose,  including  alignment  with  business  objectives  (see  activity  2. 1.1. 1 
Determine  Appraisal  Objectives) 
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•  the  appraisal  reference  model  scope  (see  activity  2. 1.1.3  Determine  Appraisal  Scope) 

•  the  appraised  organization  (see  activity  2. 1. 1 .3  Determine  Appraisal  Scope) 

•  the  process  context,  which,  at  a  minimum,  includes 

-  appraised  organization  size  and  demographics 

-  application  domain,  size,  criticality,  and  complexity 

If  application  domain,  size,  criticality,  and  complexity  are  not  relevant  in  the  appraised 
organization,  then  other  characteristics  should  be  identified  in  planning  that  are  relevant 
to  the  process  context  and  business  drivers  of  organization. 

•  appraisal  constraints  (see  activity  2.1. 1.2  Determine  Appraisal  Constraints),  which,  at  a 
minimum,  include  the  following: 

-  availability  of  key  resources  (e.g.,  staffing,  funding,  tools,  facilities) 

-  schedule  constraints 

-  the  maximum  amount  of  time  to  be  used  for  the  appraisal 

-  specific  PAs  or  organizational  entities  to  be  excluded  from  the  appraisal 

-  the  maximum,  minimum,  or  specific  sample  size  or  coverage  desired  for  the 
appraisal 

-  ownership  of  appraisal  results  and  any  restrictions  on  their  use 

-  controls  on  information  resulting  from  a  confidentiality  agreement 

-  non-attribution  of  appraisal  data  to  associated  sources 

•  the  identity  and  affiliation  of  the  Lead  Appraiser  who  is  to  be  the  appraisal  team  leader 

•  the  identity  and  affiliation  of  the  appraisal  team  members,  with  their  specific  appraisal 
responsibilities 

•  the  identity  (name  and  organizational  affiliation)  of  appraisal  participants  and  support 
staff  and  their  specific  responsibilities  for  the  appraisal 

•  any  additional  information  to  be  collected  during  the  appraisal  to  support  the 
achievement  of  the  appraisal  objectives 

•  a  description  of  the  planned  appraisal  outputs  (see  activity  2. 1.1. 4  Determine  Outputs), 
including  ratings  to  be  generated 

•  anticipated  follow-on  activities  (e.g.,  reports,  appraisal  action  plans,  reappraisal) 

•  planned  tailoring  of  SCAMPI  with  People  CMM  (see  activity  2. 1 .2. 1  Tailor  Method)  and 
associated  tradeoffs,  including  the  sample  size  or  coverage  of  the  appraised  organization, 
as  permitted  by  SCAMPI  with  People  CMM 

•  appraisal  usage  mode  (i.e.,  Internal  Process  Improvement  or  Process  Monitoring) 

Optional  Practices — None. 

Implementation  Guidance — A  Lead  Appraiser’s  ability  to  build  and  maintain  commitment 
from  the  sponsor  and  the  members  of  the  sponsoring  organization  is  a  major  factor 
contributing  to  the  success  of  the  appraisal.  The  process  of  understanding  the  requirements 
and  constraints  should  yield  a  series  of  agreements  that  form  an  input  to  the  appraisal  plan 
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(i.e.,  the  appraisal  plan  might  evolve  through  controlled  versions  as  agreements  between  the 
sponsor  and  appraisal  team  leader  reflect  changes).  Based  on  the  judgment  of  the  appraisal 
team  leader,  these  agreements  might  be  covered  in  a  formal,  signed  document  that  forms  a 
basis  for  future  activities.  More  typically,  the  appraisal  team  leader  maintains  a  record  of 
interactions  with  the  sponsor,  which  are  incorporated  into  the  appraisal  plan  as  it  is  drafted. 

The  appraisal  team  leader  and  the  sponsor  should  reach  verbal  agreement  on  the  items 
discussed  above,  and  these  items  should  be  documented  in  some  way.  The  formality  of  the 
documentation  might  range  from  simple  meeting  minutes  maintained  by  the  appraisal  team 
leader,  to  a  more  formal  memorandum  of  understanding  or  other  vehicle  that  documents 
agreements  and  provides  traceability.  The  appraisal  plan  is  used  to  document  important  issues 
pertaining  to  requirements. 

2.1 .2  Develop  Appraisal  Plan 

Purpose — Document  the  results  of  appraisal  planning,  including  the  requirements, 
agreements,  estimates,  risks,  method  tailoring,  and  practical  considerations  (e.g.,  schedules, 
logistics,  and  contextual  information  about  the  organization)  associated  with  the  appraisal. 
Obtain  and  record  the  sponsor’s  approval  of  the  appraisal  plan. 

Entry  Criteria— An  appraisal  sponsor  and  SCAMPI  with  People  CMM  Lead  Appraiser  have 
agreed  to  proceed  with  appraisal  planning,  based  on  a  common  understanding  of  the  key 
parameters  that  drive  the  planning  process. 

All  required  information  is  captured  in  the  appraisal  plan,  including  changes,  once  the  plan 
has  been  baselined.  There  can  be  evolving  versions  of  the  appraisal  plan  and  new  versions 
will  also  be  baselined. 

Inputs — Documented  agreements,  reflected  in  the  appraisal  input,  that  support  a  common 
understanding  of  appraisal  objectives  and  key  appraisal-planning  parameters. 

Activities — The  six  activities  listed  below  are  required  for  this  process. 

2. 1 .2. 1  Tailor  Method 

2. 1.2.2  Identify  Needed  Resources 

2. 1.2.3  Determine  Cost  and  Schedule 

2. 1.2.4  Plan  and  Manage  Logistics 

2. 1.2.5  Document  and  Manage  Risks 

2. 1.2.6  Obtain  Commitment  to  Appraisal  Plan 

Outputs 

•  approved  appraisal  plan 

•  strategy  for  managing  logistics 
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•  strategy  for  preparing  the  organization(s) 

•  schedule 

•  interview  plan 

•  team  assignments 

•  survey  plan 

Outcome — The  sponsor  and  appraisal  team  leader  agree  on  technical  and  non-technical 
details  for  the  planned  appraisal.  The  plan  is  refined  in  conjunction  with  the  other  Planning 
and  Preparation  phase  activities.  This  agreement  is  documented  and  reviewed  by  affected 
stakeholders  as  appropriate. 

Exit  Criteria — The  final  appraisal  plan  is  reviewed  and  approved. 

Key  Points — Skilled  appraisal  team  leaders  effectively  develop  and  use  outputs  from  the 
other  Planning  and  Preparation  phase  activities  to  achieve  the  clarity  of  shared  vision 
necessary  to  make  the  tradeoffs  and  decisions  resulting  in  a  final  plan.  This  activity  is  an 
important  opportunity  for  the  appraisal  team  leader  to  demonstrate  process  discipline  and  the 
type  of  careful  planning  described  in  the  People  CMM  model.  Experienced  appraisal  team 
leaders  are  able  to  leverage  data,  templates,  and  assets  (developed  through  their  own 
experience)  to  improve  the  completeness  and  effectiveness  of  the  appraisal  plan,  recognizing 
the  return  on  investment  that  is  obtained  through  smooth  and  efficient  appraisals. 

Tools  and  Techniques — Tools  include  an  appraisal  plan  template,  samples,  and  embedded 
procedural  guidance  in  planning  templates.  Estimation  worksheets  and  methods  for  assessing 
the  impact  of  appraisal  constraints  are  also  quite  useful. 

Metrics 

•  calendar  time  spanned  by  the  activity 

•  effort  consumed  in  carrying  out  the  activities  of  this  process 

•  level  and  frequency  of  changes  to  the  appraisal  plan 
Metrics  are  captured  in  the  appraisal  plan. 

Verification  and  Validation 

•  comparison  of  actual  effort  for  this  activity  with  historical  data  accumulated  by  the 
appraisal  team  leader 

•  review  of  the  appraisal  plan  by  affected  stakeholders 

•  sponsor’s  approval  of  the  plan 
■Records 

•  estimation  worksheets  (if  used) 
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•  appraisal  plan  (for  a  detailed  list  of  plan  contents,  see  activity  2. 1 .2.6  Obtain 
Commitment  to  Appraisal  Plan) 

Tailoring— In  some  applications,  templates  and  procedures  in  routine  use  within  the 
organization  can  be  adapted  to  the  needs  of  the  appraisal.  This  aids  in  communication  as  well 
as  local  ownership  of  the  process. 

A  structured  planning  workshop  might  benefit  organizations  with  limited  appraisal 
experience.  Such  a  workshop  is  a  valuable  opportunity  to  discover  risks  and  develop 
mitigation  strategies. 

Interfaces  with  Other  Processes — The  appraisal  plan  will  guide  and  define  the  execution  of 
the  appraisal  to  make  it  consistent  with  business  needs  and  constraints.  An  initial  plan  can  be 
generated  immediately  following  consultation  with  the  sponsor.  Further  refinement  is  done  as 
detailed  planning  occurs  and  new  information  comes  to  light  during  appraisal  planning  and 
preparation.  A  final  appraisal  plan  must  be  completed  prior  to  the  completion  of  process  2. 1 .5 
Prepare  for  Collection  of  Objective  Evidence.  Typically,  resources,  method  tailoring,  model- 
related  decisions,  and  a  planned  list  of  outputs  are  finalized  early  on,  while  cost,  schedule, 
and  logistics  are  finalized  later  while  planning  and  preparing  for  the  appraisal  process.  When 
a  combined  SCAMPI  and  SCAMPI  with  People  CMM  appraisal  is  planned,  one  combined 
plan  should  be  completed. 

The  appraisal  input  is  necessary  to  the  appraisal-planning  process.  While  it  might  not  be 
necessary  to  formally  separate  the  requirements  analysis  activities  from  the  activities 
described  in  this  section,  prior  understanding  of  the  appraisal  requirements  is  a  necessary 
input  to  this  process.  The  plan  for  the  appraisal  provides  an  important  vehicle  for  the 
following: 

•  documenting  agreements  and  assumptions 

•  establishing  and  maintaining  sponsorship 

•  tracking  and  reporting  the  performance  of  the  appraisal  process 

•  reinforcing  commitments  at  key  points  in  the  appraisal  process 

The  distinction  between  the  appraisal  input  and  appraisal  plan  is  intended  to  separate  the  key 
appraisal  requirements  and  strategic  objectives,  which  require  high  sponsor  visibility  and 
change  control  approval,  from  the  tactical  planning  details  necessary  to  implement  and  satisfy 
these  objectives.  While  sponsor  visibility  into  the  appraisal  plan  is  necessary,  revisions  are 
typically  low-level  implementation  details  and  do  not  ordinarily  require  sponsor  re-approval. 
In  practical  use,  the  appraisal  input  is  often  packaged  as  a  component  of  the  appraisal  plan, 
and  a  single  sponsor  signature  can  serve  as  approval  for  both. 

Summary  of  Activities — This  process  is  composed  of  six  activities  summarized  below.  The 
scope  of  the  appraisal  is  defined  in  terms  of  the  portion  of  the  People  CMM  model  that  is 
investigated  and  the  bounds  of  the  appraised  organization  for  which  the  results  can  be 
considered  valid  (e.g.,  a  unit,  a  product  line,  an  operating  division,  a  business  unit,  an  entire 
global  enterprise). 
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Method-tailoring  choices  are  made  to  achieve  most  effectively  appraisal  objectives  within 
defined  constraints  of  time,  effort,  and  cost.  The  resources  required  to  carry  out  the  appraisal 
are  identified.  The  cost  and  schedule  are  negotiated  and  recorded.  The  details  of  logistics, 
particularly  for  the  on-site  period,  are  documented.  Risks  and  risk-mitigation  plans  are 
identified  and  documented.  Completion  of  these  activities  results  in  a  well-defined, 
achievable  appraisal  plan. 

2.1. 2.1  Tailor  Method 

Activity  Description — Tailoring  of  SCAMPI  with  People  CMM  can  include  the  following: 

•  selection  of  choices  (if  any)  within  the  required  practices  as  defined  in  this  interpretation 
guide  for  this  appraisal  method 

•  setting  parameters  that  are  allowed  to  vary  within  the  Parameters  and  Limits 

•  inclusion  of  Optional  Practices  defined  in  this  interpretation  guide  for  this  appraisal 
method 

Because  SCAMPI  with  People  CMM  is  designed  to  apply  to  a  wide  range  of  appraisal 
applications,  the  tailoring  activity  is  one  that  deserves  careful  and  thoughtful  attention. 

Using  “partially  satisfied”  and  choosing  to  do  the  appraisal  in  “verification”  or  “discovery” 
mode  are  explicit  tailoring  options.  This  plan  is  designed  to  clearly  indicate  which  aspects  of 
the  method  are  required  and  which  are  tailorable.  The  Parameters  and  Limits  and  Optional 
Practices  sections  of  each  activity  description  provide  discussions  of  tailoring  options  in 
context.  In  addition,  the  appraisal  usage  mode  will  determine  some  tailoring  choices. 

Required  Practices 

•  Review  and  select  tailoring  options  within  the  required  practices  in  each  activity  defined 
in  this  appraisal  method. 

•  Review  and  set  parameters  within  acceptable  limits  where  variation  is  expected. 

•  Review  and  select  appropriate  optional  practices. 

•  Ensure  that  the  tailoring  decisions  are  self-consistent  and  that  they  are  appropriate  in  light 
of  the  appraisal  objectives  and  constraints. 

•  Document  the  tailoring  decisions  made. 

All  tailoring  and  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — The  structure  of  the  SCAMPI  with  People  CMM  interpretation 
guide  clarifies  which  SCAMPI  with  People  CMM  features  are  required,  either  as  a  direct 
derivative  of  ARC  requirements  or  as  SCAMPI  with  People  CMM  requirements.  Parameters 
and  Limits  sections  define  the  allowable  variation  within  these  method  requirements. 
Tailoring  guidance  and  implementation  guidance  are  provided  to  assist  with  tuning  the 
method  to  fit  sponsor  objectives  and  appraisal  constraints.  Method  tailoring  and 
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implementation  options  must  be  selected  and  implemented  in  a  way  that  does  not  violate 
SCAMPI  with  People  CMM  required  practices. 

No  tailoring  should  be  made  that  violates  ARC  or  SCAMPI  with  People  CMM  required 
practices.  All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Optional  Practices — Provide  the  sponsor  with  more  than  one  candidate  scenario  for  the 
appraisal,  and  help  them  select  from  the  options.  Alternatively,  the  appraisal  team  leader 
could  define  a  tailored  instance  of  the  method  and  propose  it  to  the  sponsor  for  approval  or  to 
negotiate  some  of  the  details. 

Implementation  Guidance— This  appraisal  method  offers  a  wide  variety  of  choices  that 
allow  the  appraisal  team  leader  and  sponsor  to  select  appraisal  features  that  best  address 
appraisal  and  business  objectives.  The  SCAMPI  with  People  CMM  interpretation  guide  is  an 
asset  provided  to  Lead  Appraisers  by  the  People  CMM  Steward  who  assists  with 
understanding  SCAMPI  with  People  CMM  tailoring  and  implementation  choices. 

Method  tailoring  is  directly  related  to  organizational  scope  and  model  scope  decisions.  Most 
of  the  allowable  tailoring  options  flow  logically  from  these  decisions  when  taken  in  context 
of  the  appraisal  objectives  and  constraints.  Tailoring  decisions  typically  affect  the  appraisal 
risk. 

Typical  tailoring  choices  that  significantly  impact  appraisal  planning  include  the  following: 

•  People  CMM  model  PAs  encompassed  by  the  appraisal 

•  specification  of  the  organization  to  be  appraised 

•  number,  experience,  skills,  and  affiliation  (e.g.,  internal  or  external)  of  appraisal  team 
members 

•  data  collection,  analysis,  and  validation  approaches  to  be  used,  including  supporting  work 
aids  and  tools 

•  effort  invested  by  the  organization  and  the  appraisal  team  in  preparation,  including  pre- 
on-site  data  collection  and  analysis 

•  time  spent  on  site 

•  choice  of  models  to  be  appraised  (See  Appendix  I  for  combined  appraisals.) 

Experienced  appraisal  team  leaders  provide  a  well-defined  approach  to  ensure  that  the 
appraisal  objectives  are  achieved  in  an  efficient  and  effective  manner.  Experienced  sponsors 
require  a  well-defined  approach  to  ensure  an  acceptable  level  of  risk  in  meeting  objectives 
within  the  constraints.  The  appraisal  plan  documents  the  method-tailoring  decisions  and  their 
rationale,  and  the  associated  method  variations  and  techniques  that  will  be  employed. 
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2.1. 2.2  Identify  Needed  Resources 

Activity  Description — This  activity  involves  the  identification  and  estimation  of  resources 
needed  to  carry  out  the  appraisal.  Resources  include  personnel,  facilities,  tools,  and  access  to 
information. 

Required  Practices 

•  Identify  appraisal  team  members. 

•  Identify  appraisal  participants. 

•  Identify  equipment  and  facilities. 

•  Identify  other  appraisal  resources  needed. 

•  Document  resource  decisions  in  the  appraisal  plan. 

•  Select  and  invite  survey  and  appraisal  interview  participants.  Activity  2. 1 .4. 1  Plan  Survey 
Administration  describes  selecting  survey  participants.  On-site  appraisal  participant 
selection  is  described  in  activity  2.1.5.2  Prepare  Data  Collection  Plan. 

•  Identify  documents  and  forms  required  for  review. 

Parameters  and  Limits — The  level  of  detail  in  the  identification  of  needed  resources  must 
be  sufficient  to  support  the  creation  of  the  appraisal  plan.  For  example,  the  appraisal  team 
leader  must  identify  the  following: 

•  the  names  of  people  who  are  candidates  for  interviews  or  appraisal  team  membership 

•  the  organizational  or  unit  affiliation  of  these  people 

•  the  location,  seating  capacity,  and  configuration  of  rooms  to  be  used  by  the  team 

•  specific  equipment  needed  (e.g.,  overhead  projector,  laptop  projector,  video- 
conferencing) 

All  required  information  is  captured  in  the  appraisal  plan. 

Optional  Practices — Several  months  before  the  appraisal,  tour  the  facility  where  the 
appraisal  will  be  held.  Assign  an  individual  from  the  appraised  organization  to  carry  out  the 
duties  of  the  appraised  organization  coordinator  (e.g.,  administrative  and  logistical  support; 
see  activity  2. 1.3.2  Select  Team  Members). 

Implementation  Guidance — Appraisal  resources  are  typically  defined  early  in  the  appraisal¬ 
planning  process.  Identifying  resources  goes  hand  in  hand  with  estimating  appraisal  cost  and 
schedule  (see  activity  2. 1.2.3  Determine  Cost  and  Schedule),  and  these  might  be  iteratively 
refined.  Tradeoffs  are  routinely  made  in  light  of  the  appraisal  objectives  and  constraints. 

The  appraisal  sponsor  or  senior  site  manager  might  identify  candidate  appraisal  team 
members  and  appraisal  participants.  Review  of  the  appraised  organization  structure  or  other 
site-specific  information  can  also  be  useful  for  this.  Initially,  participants  can  be  specified  in 
terms  of  roles  or  responsibilities,  with  names  to  be  determined  later.  Process  2.1.3  Select  and 
Prepare  Team  contains  additional  guidance  on  selecting  appraisal  team  members. 
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Equipment  and  facilities  are  often  negotiated  with  the  appraised  organization  where  the 
appraisal  on-site  activities  are  performed,  but  sometimes  these  must  be  acquired.  A  room  for 
dedicated  use  by  the  appraisal  team  is  usually  necessary  for  private  discussions  and  to  protect 
the  confidentiality  of  appraisal  data.  Ideally,  this  is  separate  from  the  rooms  where  interview 
sessions  are  held. 

The  availability  of  computing  resources,  such  as  computers,  printers,  and  networks,  is  a  key 
consideration  that  should  be  planned  and  understood.  Access  to  special  tools  or  applications 
might  also  be  needed. 

2.1. 2.3  Determine  Cost  and  Schedule 

Activity  Description — A  top-level  cost  breakdown  and  schedule  are  developed  and  included 
in  the  plan. 

Required  Practices 

•  Estimate  the  duration  of  key  events  as  a  basis  for  deriving  a  comprehensive  schedule. 

•  Estimate  the  effort  required  for  the  people  participating  in  the  appraisal. 

•  Estimate  the  costs  associated  with  using  facilities  and  equipment  (as  appropriate). 

•  Estimate  the  costs  for  incidentals  (e.g.,  travel,  lodging,  meals)  as  appropriate. 

•  Document  detailed  schedule  estimates  in  the  appraisal  plan. 

•  Document  detailed  cost  estimates  in  the  appraisal  plan. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits— Effort  estimates  should  be  developed  for  the  appraisal  team  and 
the  expected  participants  within  the  appraised  organization  (e.g.,  interviewees,  respondents  to 
instruments  administered,  attendees  at  briefings,  support  staff).  Scheduling  for  each  day  of 
the  appraisal  is  required. 

Optional  Practices — A  preliminary  daily  schedule  could  be  determined  for  the  first  baseline 
of  the  appraisal  plan  and  built  upon  from  the  appraisal  input. 

Implementation  Guidance — Cost  and  schedule  might  be  developed  top  down  based  on 
sponsor  objectives  and  constraints,  bottom  up  based  on  results  of  other  planning  and 
preparation  processes  and  activities,  or  more  generally  using  a  combination  of  the  two 
approaches.  Scheduling  the  events  and  activities  of  the  appraisal  is  an  ongoing  logistical  task 
that  requires  the  coordination  of  many  different  groups.  Determining  and  communicating  a 
schedule  for  the  appraisal  and  maintaining  ongoing  visibility  as  the  details  take  form  are  the 
primary  responsibilities  of  the  appraisal  team  leader.  The  appraised  organization  coordinator 
is  expected  to  provide  support  for  this  task,  and  the  appraisal  team  leader  typically  selects  the 
person  who  plays  that  role  with  this  duty  in  mind. 
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The  sponsor’s  needs  for  appraisal  outputs  of  a  specified  quality  to  fulfill  a  specific  purpose, 
balanced  against  the  resources  available  to  conduct  the  appraisal,  will  determine  the  schedule 
constraints.  Schedule  and  cost  need  to  be  considered  for  the  entire  span  of  appraisal  activities. 
The  tradeoff  between  time  spent  in  preparation  versus  time  spent  on  site  will  therefore  be  a 
significant  factor,  as  will  post-on-site  reporting  activities. 

Organizational  costs  for  preparing  and  supporting  appraisals  can  be  reduced  by  gathering  and 
maintaining  objective  evidence  for  each  unit  instance.  In  addition  to  providing  an  effective 
mechanism  for  monitoring  the  process  implementation  and  improvement  progress  of  each 
unit,  this  enables  the  availability  and  reuse  of  objective  evidence  for  subsequent  appraisals. 

While  the  schedule  for  the  appraisal  is  usually  shared  with  a  fairly  wide  audience,  the  cost  of 
the  appraisal  and  some  elements  within  the  appraisal  are  often  kept  from  wide  view  due  to 
the  potentially  sensitive  nature  of  this  information. 

2.1. 2.4  Plan  and  Manage  Logistics 

Activity  Description— The  logistical  details  of  the  on-site  portion  of  the  appraisal  are 
negotiated  and  documented.  The  appraisal  team  leader,  supported  by  the  appraised 
organization  coordinator,  manages  planning  tasks  that  document  and  communicate  logistical 
arrangements.  Checklists  and  action  item  tracking  mechanisms  are  very  important  structures 
used  to  manage  these  tasks. 

Required  Practices 

•  Document  logistical  schedules  and  dependencies. 

•  Maintain  communication  channels  for  providing  status. 

•  Assign  responsibilities  for  tracking  logistical  issues. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits 

Plan  the  logistics  to  support  the  following  appraisal  tasks  and  activities,  if  not  previously 
provided,  including  the  following: 

•  all  remaining  preparation  tasks,  including  team  training 

•  survey  administration  and  processing 

•  on-site  appraisal  tasks 

•  on-site  logistics,  such  as  times,  rooms,  food,  security,  computer  access,  printers, 
secretarial  support,  and  audio-visual  equipment 

•  report  preparation 

•  team  travel  and  lodging 

•  initial  post-appraisal  improvement  activities 
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•  assignment  and  orientation  of  legal  representation  to  review  preliminary  findings 

Effective  planning  depends  on  anticipating  a  variety  of  logistical  issues  that  might  occur 
during  the  appraisal.  Issues  that  are  sometimes  overlooked  include  the  following: 

•  identifying  hotels  for  people  traveling  to  the  appraisal 

•  providing  workstation  support 

•  ordering  meals 

•  interacting  with  facilities  staff  on  site 

•  meeting  security  and  classification  requirements 

•  providing  badges  or  arranging  for  escorts  in  limited-access  facilities 
Optional  Practices — Assign  improvement  program  responsibilities. 

Implementation  Guidance — Every  experienced  appraisal  team  leader  knows  the  value  of 
thorough  logistical  planning  and  tracking.  The  time-critical  nature  of  on-site  appraisal 
activities  makes  it  very  difficult  to  manage  last-minute  changes  in  important  details  such  as: 

•  availability  of  conference  rooms  and  meeting  rooms  of  the  appropriate  size 

•  access  to  rooms,  equipment,  and  supplies  needed  for  administrative  tasks 

•  transportation  and/or  lodging  for  team  members  or  the  remote  members  of  the  appraised 
organization 

•  food  and  other  amenities  required  for  adequate  working  conditions 

•  communication  channels  and  back-up  staff  to  support  the  team  on  site 

2.1. 2.5  Document  and  Manage  Risks 

Activity  Description — As  with  any  unit  containing  dependencies  among  events,  people,  and 
other  resources,  risk  management  is  important  to  success.  The  appraisal  team  leader  is 
responsible  for  documenting  and  communicating  risks  and  associated  mitigation  plans  to  the 
sponsor  and  appraisal  team  members. 

Required  Practices 

•  Identify  appraisal  risks,  including  organizational  readiness  for  the  appraisal  and 
improvement. 

•  Develop  mitigation  plans  for  key  appraisal  risks  and  implement  these  plans  as  necessary. 

•  Keep  the  appraisal  sponsor  and  other  stakeholders  informed  of  the  appraisal  risk  status. 
Risks  are  defined  and  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — The  risks  and  mitigation  plans  identified  during  this  activity  are 
required  elements  of  the  appraisal  plan  (see  Parameters  and  Limits  for  activity  2. 1.2.6  Obtain 
Commitment  to  Appraisal  Plan).  Most  Lead  Appraisers  include  a  section  titled  “Risk 


CMU/SEI-2005-SR-001 


23 


Management”  in  the  appraisal  plan.  The  level  of  effort  devoted  to  risk-management  activities 
is  something  the  appraisal  team  leader  must  adjust  to  fit  the  situation  at  hand. 

Optional  Practices — Risks  regarding  the  appraisal  team  should  be  included  in  some 
situations. 

Implementation  Guidance — The  appraisal  plan  is  used  to  document  and  track  risks  to  the 
successful  conduct  of  the  appraisal.  As  with  the  requirement  to  address  logistical  issues 
during  planning,  there  are  no  minimum  guidelines  to  be  met  other  than  that  the  plan  include 
identified  risks  and  planned  mitigation  strategies. 

The  appraisal  team  leader  is  responsible  for  keeping  the  appraisal  sponsor  informed  of  risk- 
management  activities  so  that,  if  needed,  timely  sponsor  intervention  is  possible  to  ensure  the 
achievement  of  appraisal  objectives. 

2.1. 2.6  Obtain  Commitment  to  Appraisal  Plan 

Activity  Description — Formal  sponsor  commitment  to  the  appraisal  plan  is  obtained.  The 
appraisal  plan  constitutes  a  “contract”  between  the  appraisal  sponsor  and  the  appraisal  team 
leader,  so  it  is  vital  that  this  agreement  be  formal. 

Required  Practices 

•  Document  the  appraisal  plan. 

•  Review  the  appraisal  plan  with  the  sponsor  and  secure  the  sponsor’s  approval. 

•  Provide  the  appraisal  plan  to  relevant  stakeholders  for  review. 

•  Ensure  the  involvement  of  human  resources  and  other  related  process  owners. 

•  Identify  other  stakeholders  affected  by  the  appraisal  scope  and  ensure  their  involvement. 

•  Obtain  commitment  of  participation  from  the  assessed  population. 

The  appraisal  plan  is  signed  by  the  Lead  Appraiser  and  sponsor  when  it  is  baselined. 

Parameters  and  Limits — Required  contents  of  the  appraisal  plan  include  the  following,  at  a 
minimum: 

•  the  appraisal  input  (see  activity  2. 1 . 1 .5  Obtain  Commitment  to  Appraisal  Input) 

•  the  activities  to  be  performed  in  conducting  the  appraisal 

•  resources  needed  for  conducting  the  appraisal  (see  activity  2. 1.2.2  Identify  Needed 
Resources) 

•  cost  and  schedule  estimates  for  performing  the  appraisal  (see  activity  2. 1.2.3  Determine 
Cost  and  Schedule) 

•  appraisal  logistics  (see  activity  2. 1.2.4  Plan  and  Manage  Logistics) 

•  risks  and  mitigation  plans  associated  with  appraisal  execution  (see  activity  2. 1 .2.5 
Document  and  Manage  Risks) 
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The  plan  should  also  address  the  following: 

•  documenting  appraisal  goals  and  scope 

•  identifying  the  appraisal  outputs  and  their  anticipated  use 

•  identifying  anticipated  follow-on  activities 

•  documenting  any  planned  tailoring  of  the  appraisal  method  and  associated  tradeoffs 

•  identifying  risks  associated  with  appraisal  execution 

There  must  be  a  signature  block  for  the  appraisal  team  leader  and  the  sponsor  to  indicate  their 
commitment  to  the  plan  in  writing.  If  minor  updates  are  made  to  the  plan,  signatures  do  not 
have  to  be  obtained  again  unless  one  or  more  elements  of  the  appraisal  input  have  been 
changed.  At  a  minimum,  the  appraisal  team  members  are  considered  relevant  stakeholders 
and  should  receive  a  copy  of  the  approved  appraisal  plan. 

Optional  Practices — Use  a  signature  block  for  relevant  stakeholders  to  indicate  their 
commitment  to  the  plan  in  writing  (i.e.,  each  team  member  signs  the  plan). 

Implementation  Guidance — While  sponsor  visibility  into  the  appraisal  plan  is  necessary, 
revisions  are  typically  low-level  implementation  details  and  do  not  ordinarily  require  sponsor 
re-approval.  This  is  in  contrast  to  the  appraisal  input,  which  contains  strategic,  key  appraisal 
requirements,  objectives,  and  constraints.  Revisions  to  the  appraisal  input  must  be  approved 
by  the  sponsor.  In  practical  use,  the  appraisal  input  is  often  packaged  as  a  component  of  the 
appraisal  plan,  and  a  single  sponsor  signature  can  serve  as  approval  for  both.  The  separation 
of  the  appraisal  input  and  appraisal  plan  is  intended  to  provide  an  appropriate  level  of  sponsor 
visibility  and  approval  while  leaving  appraisal  team  leaders  the  flexibility  to  refine  the  low- 
level  details  necessary  to  complete  thorough  appraisal  planning. 

The  use  of  the  term  “relevant  stakeholder”  in  the  context  of  appraisal  planning  is  intended  to 
be  interpreted  broadly  to  include  as  many  of  the  participants  and  other  affected  parties  as 
feasible. 

By  this  time  the  involvement  of  human  resources  and  other  related  process  owners  should  be 
identified  in  the  appraisal  plan  and  their  involvement  ensured.  If  there  are  other  stakeholders 
affected  by  the  appraisal  scope  they  should  be  identified  in  the  appraisal  plan  and  their 
involvement  ensured.  Obtain  commitment  to  participate  from  the  assessed  population  by 
keeping  them  informed  of  the  appraisal  plan  and  the  organization’s  objectives  for  the 
appraisal. 

2.1 .3  Select  and  Prepare  Team 

Purpose — Ensure  that  an  experienced,  trained,  appropriately  qualified  team  is  available  and 
prepared  to  execute  the  appraisal  process. 
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Entry  Criteria 

•  Appraisal  requirements  have  been  documented,  at  least  in  draft  form. 

•  Appraisal  constraints  are  understood  and  documented,  at  least  in  draft  form. 

•  The  appraisal  plan  is  defined,  at  least  in  draft  form. 

•  The  appraisal  implementation  method  has  been  identified  (see  activity  2.1. 1.3  Determine 
Appraisal  Scope). 

Inputs 

•  appraisal  requirements  and  constraints  (in  draft  or  final  form) 

•  appraisal  plan  (in  draft  or  final  form) 

•  Team  training  materials  for 

-  People  CMM 

-  SCAMPI  when  a  combined  SCAMPJTPeople  CMM  appraisal 
Activities— The  three  activities  listed  below  are  required  for  this  process. 

2. 1 . 3 . 1  Identify  Team  Leader 

2. 1.3.2  Select  Team  Members 

2. 1.3.3  Prepare  Team 

Outputs 

•  training  records 

•  appraisal  team  member  assignments  and  qualifications 

•  a  prepared  appraisal  team  that  has  completed 

-  appraisal  method(s)  training 

-  reference  model(s)  training 

-  team-building  activities 

-  team  orientation  regarding  appraisal 

Outcome 

The  successful  completion  of  this  process  results  in  an  experienced,  trained,  and  oriented 
team  ready  to  execute  the  appraisal  for  either  a  stand-alone  SCAMPI  with  People  CMM  or  a 
combined  SCAMPI  with  CMMI  and  People  CMM.  The  appraisal  team  members  have 
acquired  the  necessary  knowledge  to  play  their  roles,  or  their  previous  knowledge  is 
determined  to  be  satisfactory.  The  appraisal  team  leader  has  provided  opportunities  to 
practice  the  skills  needed  for  each  person  to  play  his  or  her  role  or  has  confirmed  that  these 
skills  have  already  been  demonstrated  in  the  past.  The  team  members  have  been  introduced  to 
.one  another  and  have  begun  to  plan  how  they  will  work  together. 

Exit  Criteria 

•  The  prepared  team  is  committed  to  the  appraisal. 
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•  Training  has  been  provided  and  its  results  recorded. 

•  Remediation  of  knowledge/skill  shortfalls  has  been  completed  (if  needed). 

Key  Points —  The  appraisal  team  leader  is  responsible  for  ensuring  that  the  team  is  ready  to 
succeed,  whether  the  appraisal  team  leader  trains  an  intact  team  or  forms  a  team  from  a  corps 
of  experienced  team  members. 

Tools  and  Techniques — Training  course  material  is  available  from  the  People  CMM  Steward 
and  the  CMMI  Steward  for  training  teams.  This  should  be  tailored  or  supplemented  by  the 
appraisal  team  leader  based  on  the  appraisal  context  or  degree  of  team  member  experience. 
Case  studies  and  exercises  are  recommended  to  reinforce  the  situations  team  members  are 
likely  to  encounter  during  the  appraisal. 

Other  ways  of  accomplishing  this  activity  might  draw  on  one  or  more  of  the  following: 

•  providing  supplementary  training  to  experienced  team  members  so  that  the  operational 
details  of  the  approach  used  will  be  familiar 

•  training  a  cadre  of  team  members  and  keeping  their  knowledge  and  skills  up-to-date  as 
part  of  an  overall  program  of  appraisals 

Metrics 

•  summary  of  team  member  qualifications 

•  effort  and  calendar  time  expended  to  accomplish  training 

•  trainee  ratings  of  instructional  materials  and  approach  (if  applicable) 

•  achievement  of  milestones  for  remedial  activities  (if  applicable) 

The  appraisal  plan  will  delineate  each  appraisal  team  member’s  qualifications. 

Metrics  are  captured  in  the  appraisal  plan. 

Verification  and  Validation 

•  approval  of  team  membership  and  preparation  by  the  sponsor  and  appraisal  team  leader 

•  results  of  exams  used  to  demonstrate  training  effectiveness  (if  used) 

•  feedback  from  team  members  on  their  readiness  to  perform  their  roles 

Records 

•  team  member  contact  information 

•  training  records  (if  applicable) 

•  feedback  provided  by  trainees  (if  applicable) 

•  team  qualification  summary  (recorded  in  appraisal  plan) 
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Tailoring 

•  Case  study  materials  provide  a  variety  of  options  for  expanding  the  team  training 
course(s)  to  add  emphasis  where  desired. 

•  Experienced  appraisal  team  leaders  have  had  success  conducting  role-plays  and 
simulated  appraisal  activities  without  case  studies. 

•  When  assembling  a  team  of  already-trained  members,  conduct  team-building  activities  to 
ensure  team  cohesion.  Many  team-building  exercises  are  available  for  this  purpose. 

•  Team  size,  skills,  and  composition  are  tailoring  options  in  the  method. 

Interfaces  with  Other  Processes — This  process  includes  selecting  and  preparing  the 
appraisal  team.  It  might  occur  after  obtaining  sponsor  commitment  to  the  appraisal  input.  The 
appraisal  plan  should  be  available,  at  least  in  draft  form,  as  a  necessary  input  (for  contents  see 
activity  2. 1.2.6  Obtain  Commitment  to  Appraisal  Plan).  Selected  appraisal  team  members 
might  provide  input  into  further  definition  of  the  appraisal  planning.  Appraisal  team  training 
might  provide  an  initial  means  to  obtain  a  preliminary  understanding  of  the  appraised 
organization’s  operations  and  processes.  If  available,  the  appraised  organization’s  PII 
database  from  a  previous  People  CMM  appraisal  or  People  CMM  mini-appraisal  is  a  useful 
resource  for  orienting  the  appraisal  team  on  organizational  characteristics,  such  as  the 
application  domain,  the  organizational  structure,  the  process  improvement  structure,  and 
approaches  for  reference  model  implementation. 

Training  in  SCAMPI  is  only  required  when  appraisal  team  members  will  participate  in  both 
SCAMPI  with  CMMI  and  SCAMPI  with  People  CMM  appraisal  activities. 

Summary  of  Activities— The  appraisal  team  is  a  cohesive  unit  of  trained  and  capable 
professionals  who  must  meet  stringent  qualifications.  An  appraisal  team  leader  is  selected  to 
plan  and  manage  the  performance  of  the  appraisal,  delegate  appraisal  tasks  to  team  members, 
and  ensure  adherence  to  SCAMPI  with  People  CMM  requirements.  Appraisal  team  members 
are  selected  based  on  defined  criteria  for  experience,  knowledge,  and  skills  to  ensure  an 
efficient  team  capable  of  satisfying  the  appraisal  objectives.  Training  is  provided  to  ensure 
proficiency  in  the  reference  model  and  appraisal  method. 

2.1. 3.1  Identify  Team  Leader 

Activity  Description— The  appraisal  sponsor  is  responsible  for  selecting  an  appraisal  team 
leader  who  has  the  appropriate  experience,  knowledge,  and  skills  to  take  responsibility  for 
and  lead  the  appraisal.  By  definition,  an  appraisal  team  leader  must  be  a  SCAMPI  with 
People  CMM  Lead  Appraiser  authorized  by  the  SEI  Appraisal  Program  and  a  member  of  that 
program  in  good  standing.2  The  appraisal  team  leader  is  responsible  for  ensuring  that  the 
appraisal  is  conducted  in  accordance  with  SCAMPI  with  People  CMM  requirements,  with 

2  The  SEI  People  CMM  model  and  Appraisal  Program  are  described  on  the  SEI  Web  site  at 
http://www.sei.cmu.edu/cmm-p/version2  and  http://www.sei.cmu.edu/cmmi/appraisals. 
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tailoring  to  meet  appraisal  objectives  and  constraints  within  allowable  bounds  defined  by  the 
method. 

The  appraisal  team  leader  must  also  be  a  SCAMPI  Lead  Appraiser  when  a  combined 
appraisal  is  performed. 

Required  Practices 

•  Select  an  authorized  SCAMPI  with  People  CMM  Lead  Appraiser  to  serve  as  the 
appraisal  team  leader. 

•  Select  an  authorized  SCAMPI  Lead  Appraiser  to  serve  as  the  appraisal  team  leader  when 
a  combined  appraisal  is  to  be  performed. 

•  Verify  the  qualifications  of  the  appraisal  team  leader  (experience,  knowledge,  and  skills). 
All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — The  appraisal  team  leader  must  be  an  SEI-authorized  People 
CMM  Lead  Assessor  and  a  SCAMPI  Lead  Appraiser  in  good  standing.  These  can  be  verified 
on  the  Web  or  by  contacting  the  SEI  People  CMM  Steward  or  SEI CMMI  Steward  directly. 
There  can  be  only  one  official  appraisal  team  leader  on  any  given  appraisal. 

The  appraisal  team  leader  has  sole  discretion  to  delegate  important  tasks  to  appraisal  team 
members  but  cannot  delegate  leadership  responsibility  or  ultimate  responsibility  for  the 
successful  completion  of  the  appraisal.  The  inclusion  of  multiple  Lead  Appraisers  on  a  team 
for  a  given  appraisal  can  be  a  strong  asset  for  the  leader  of  that  team.  However,  the  single 
designated  appraisal  team  leader  must  perform  the  leadership  role  and  manage  the  appraisal 
process. 

Optional  Practices — In  some  uses  of  SCAMPI  with  People  CMM,  representatives  of  the 
appraisal  sponsor  might  be  delegated  a  part  of  the  appraisal  team  leader’s  responsibilities  in 
advance  of  the  initial  identification  of  an  appraisal  team  leader,  if  there  is  no  risk  to  the 
integrity  of  the  appraisal.  Examples  include  delivering  the  participant’s  briefing  and 
administering  the  survey.  This  is  typically  possible  with  higher  maturity  level  organizations. 
When  this  is  done,  responsibilities  must  be  approved  and  agreed  to  between  the  appraisal 
sponsor  and  the  Lead  Appraiser. 

Implementation  Guidance — SCAMPI  with  People  CMM  Lead  Appraisers,  by  definition, 
will  have  participated  on  a  minimum  of  two  appraisals  (one  as  an  appraisal  team  member  and 
one  as  an  observed  appraisal  team  leader).  These  requirements  are  part  of  the  SEI  Lead 
Appraiser  program.  Thereafter,  for  SCAMPI  with  People  CMM,  one  appraisal  per  year  will 
be  performed. 

When  the  appraisal  team  leader  performs  a  combined  SCAMPI  with  CMMI  and  People 
CMM  appraisal,  all  requirements  for  Lead  Appraisers  defined  by  the  SEI  for  both  People 
CMM  and  CMMI  must  be  fulfilled.  Refer  to  the  Standard  CMMI  Appraisal  Method  for 
Improvement  (SCAMPI)  Version  1.1.  Method  Definition  Document  for  details  [SEI  Ola]. 
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An  additional  consideration  that  affects  requirements  for  team  experience  is  the  maturity 
level  scope  of  the  planned  appraisal.  Additional  experience  might  be  necessary  for  the 
appraisal  team  leader  and/or  appraisal  team  members  if  the  appraisal  is  used  in  a  high- 
maturity  organization  (maturity  levels  4-5).  Special  experience,  training,  and/or  expertise 
(e.g.,  statistical  process  control)  might  be  necessary  for  that  specific  appraisal. 

Appraisal  team  leader  responsibilities  are  defined  and  described  throughout  the  SCAMPI 
with  People  CMM  interpretation  guide,  but  a  summary  overview  of  these  responsibilities 
includes  the  following: 

•  confirming  the  sponsor’s  commitment  to  proceed  with  the  appraisal 

•  ensuring  that  appraisal  participants  are  briefed  on  the  purpose,  scope,  and  approach  of  the 
appraisal 

•  ensuring  that  all  appraisal  team  members  have  the  appropriate  experience,  knowledge, 
and  skills  in  the  appraisal  reference  model  and  in  SCAMPI  with  People  CMM 

•  ensuring  that  the  appraisal  is  conducted  in  accordance  with  the  documented  SCAMPI 
with  People  CMM  method 

•  verifying  and  documenting  that  the  appraisal  method  requirements  have  been  met 

The  appraisal  team  leader  might  be  selected  at  any  time  in  the  appraisal  planning  phase — 
preferably  upon  initiation  of  appraisal  activities  so  that  he  or  she  might  participate  in 
analyzing  the  requirements  with  the  appraisal  sponsor.  In  any  event,  the  appraisal  team  leader 
should  be  identified  in  time  to  review  and  approve  the  appraisal  plan  with  the  appraisal 
sponsor  prior  to  beginning  the  on-site  portion  of  the  appraisal,  and  in  time  to  ensure  adequate 
planning  and  the  preparation  of  appraisal  team  members. 

2.1 .3.2  Select  Team  Members 

Activity  Description — This  activity  involves  identifying  available  personnel,  assessing  their 
qualifications,  and  selecting  them  as  appraisal  team  members.  It  might  occur  after  obtaining 
the  sponsor’s  commitment  to  conduct  the  appraisal  and  provide  input  to  the  appraisal 
planning. 

Required  Practices 

•  Ensure  that  minimum  criteria  for  individual  team  members  are  met. 

•  Ensure  that  minimum  criteria  for  the  team  as  a  whole  are  met. 

•  Document  the  qualifications  and  responsibilities  of  team  members  in  the  appraisal  input. 
All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

'Parameters  and  Limits — The  minimum  acceptable  team  size  for  a  SCAMPI  with  People 
CMM  appraisal  is  four  people  (including  the  team  leader). 
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All  team  members  must  have  previously  completed  the  SEI-licensed  Introduction  to  People 
CMM  course,  delivered  by  an  instructor  who  is  authorized  by  the  SEI. 

When  a  combined  SCAMPI  with  CMMI  and  People  CMM  appraisal  is  performed  all  team 
members  must  have  also  previously  completed  the  SEI-licensed  Introduction  to  CMMI 
course,  delivered  by  an  instructor  who  is  authorized  by  the  SEI. 

Team  member  training  in  the  appraisal  method  is  discussed  in  activity  2.1. 3. 3  Prepare  Team. 

The  People  CMM  appraisal  team  should  meet  specified  appraisal  team  qualification  criteria. 
Specific  team  assignment  must  include  the  following: 

•  an  SEI-authorized  People  CMM  Lead  Assessor 

•  at  least  two  members  from  the  organization  being  assessed 

•  the  improvement  team  lead(s)  or  software  engineering  process  group  (SEPG)  member(s) 
or  process  owners  most  closely  associated  with  workforce  improvement  activities.  When 
an  SEPG  leader  representative  is  selected  for  the  team,  the  appraisal  team  leader  (ATL) 
must  ensure  that  there  are  no  conflicts  of  interest.  The  ATL  always  has  the  authority  to 
have  the  SEPG  leader  representative  removed  from  the  team  if  the  integrity  of  the  results 
will  be  jeopardized. 

•  at  least  one  individual  with  human  resources  experience.  When  a  local  human  resources 
representative  is  selected  for  the  team,  the  ATL  must  ensure  that  there  are  no  conflicts  of 
interest.  The  ATL  always  has  the  authority  to  have  the  local  human  resources 
representative  removed  from  the  team  if  the  integrity  of  the  results  will  be  jeopardized. 

•  a  mix  of  backgrounds  and  job  assignments  (e.g.,  systems  engineering,  software 
engineering,  human  resources,  communications,  financial,  marketing)  representative  of 
the  primary  disciplines  or  domains  of  the  organization’s  business 

•  no  team  member  who  manages  one  of  the  selected  appraisal  units  or  is  in  the  supervisory 
chain  of  any  appraisal  participant 

•  at  least  one  team  member  should  have  a  minimum  of  five  years  experience  in  one  or 
more  of  the  following  areas: 

-  human  resources  management 

-  staffing  for  senior  positions 

-  implementing  workforce  training,  policies,  and  practices 

The  team  (as  a  group)  must  have  an  average  of  at  least  6  years  of  experience,  and  the  team 
total  must  be  at  least  25  years  of  experience  in  the  business  competencies  to  be  covered  in  the 
appraisal. 

The  team  (as  a  group)  must  have  a  total  of  at  least  10  years  of  experience,  and  at  least  one 
team  member  must  have  at  least  six  years  of  experience  as  a  manager. 

Team  members  should  not  be  managers  of  one  of  the  selected  units  or  be  within  the  direct 
supervisory  chain  of  any  of  the  anticipated  interviewees. 
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Optional  Practices — Although  not  required  in  the  Parameters  and  Limits  section  above,  the 
best  practices  listed  below  should  be  employed  whenever  feasible. 

•  Each  member  should  have  good  written  and  oral  communication  skills,  the  ability  to 
facilitate  the  free  flow  of  communication,  and  the  ability  to  perform  as  team  players  and 
negotiate  consensus. 

•  At  least  half  of  the  team  members  should  have  participated  in  a  previous  process 
appraisal. 

•  Team  members  should  be  perceived  by  the  appraisal  sponsor  as  credible. 

Additional  appraisal  team  member  selection  considerations  include  the  following: 

•  Consider  the  personal  characteristics  of  individual  team  members  (e.g.,  communication 
preferences,  personality  types)  and  how  these  might  affect  the  dynamics  in  a  team 
environment. 

•  Use  one  or  more  authorized  Lead  Appraisers  as  team  members  or  a  team  member  that 
can  serve  as  a  facilitator  in  parallel  sessions. 

Implementation  Guidance — Appraisal  team  members  should  be  a  diverse  set  of  qualified 
professionals  with  the  appropriate  experience,  knowledge,  and  skills  to  make  reasoned 
judgments  regarding  implementation  of  the  reference  model. 

The  accuracy  and  credibility  of  the  appraisal  results  depend  on  the  capability,  qualifications, 
and  preparation  of  the  appraisal  team  members.  In  addition  to  the  qualifications  described 
above,  other  factors  that  might  affect  the  performance  of  the  team  or  reliability  of  appraisal 
results  should  be  considered.  Appraisal  constraints  such  as  security  classification  might  result 
in  additional  criteria  for  team  member  selection. 

The  selected  appraisal  team  members  and  their  organizational  affiliations  and  qualifications 
(individually  and  in  aggregate)  are  documented  in  the  appraisal  plan.  Appraisal  team 
members  are  typically  selected  from  a  pool  of  qualified  individuals  provided  by  the  appraisal 
sponsor  or  his  or  her  designee.  The  appraisal  team  leader  is  the  final  authority  on  acceptance 
of  appraisal  team  members  and  is  responsible  for  ensuring  their  qualifications  and  suitability 
for  the  appraisal  purpose. 

Situations  that  present  a  conflict  of  interest  should  be  avoided.  Team  members  who  manage 
people  or  processes  in  the  organization  might  struggle  with  their  ability  to  be  objective,  and 
team  members  who  are  directly  impacted  by  the  appraisal  outcome  might  be  distracted  by  the 
potential  consequences  of  the  decisions  they  contribute  to  on  the  appraisal  team. 

2.1. 3.3  Prepare  Team 

Activity  Description — The  appraisal  team  leader  is  responsible  for  ensuring  that  appraisal 
team  members  are  sufficiently  prepared  to  perform  the  planned  appraisal  activities.  This 
includes  being  familiar  with  the  reference  model(s).  People  CMM,  CMMI,  the  appraisal  plan, 
organizational  data  and  characteristics,  and  the  tools  and  techniques  to  be  used  during  the 
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appraisal.  Roles  and  responsibilities  are  assigned  for  appraisal  tasks.  Team-building  exercises 
are  used  to  practice  facilitation  skills  and  reach  an  understanding  of  the  team  objectives  and 
how  they  will  be  satisfied. 

All  team  members  are  expected  to  observe  strict  rules  for  confidentiality,  the  protection  of 
proprietary  or  sensitive  data,  and  the  non-attribution  of  information  to  unit  participants.  Non¬ 
disclosure  or  confidentiality  statements  are  often  used  to  formalize  these  understandings. 

Required  Practices 

•  Ensure  that  appraisal  team  members  have  received  reference  model(s)  training. 

•  Provide  appraisal  method(s)  training  to  appraisal  team  members  or  ensure  that  they  have 
already  received  it. 

•  Provide  for  team  building  and  establishing  team  norms. 

•  Provide  orientation  to  team  members  on  appraisal  objectives,  plans,  and  their  assigned 
roles  and  responsibilities. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — Model  training  must  be  provided  using  the  standard  Introduction 
to  People  CMM  and  CMMI  courses  and  delivered  by  instructors  who  are  authorized  by  the 
People  CMM  Steward  and  the  CMMI  Steward,  respectively. 

Method  training  might  be  delivered  in  one  of  two  ways: 

1 .  method  training  specific  to  the  appraisal  at  hand 

2.  method  training  delivered  to  a  large  group  of  potential  future  team  members  who  are  not 
currently  engaged  in  an  appraisal 

Method  training  delivered  to  an  intact  team  must  be  at  least  two  days  in  duration  (three  days 
are  preferred)  and  must  emphasize  the  situations  likely  to  be  encountered  by  team  members 
during  the  appraisal.  This  training  will  not  necessarily  cover  all  variants  in  the  application  of 
the  SCAMPI  with  People  CMM.  For  SCAMPI  method  training,  refer  to  the  SEI  CMMI 
Steward. 

Method  training  must  cover  the  complete  set  of  tailoring  options  and  allowable  variations  for 
the  method  to  prepare  potential  team  members  for  a  range  of  situations  that  are  likely  to  arise 
during  future  appraisals.  The  SEI  Appraisal  Program  specifies  additional  requirements  for 
delivering  training  to  people  who  are  not  already  members  of  an  appraisal  team.  These  should 
be  given  consideration  and  documented  in  the  appraisal  plan  regardless  of  the  training 
decisions  made  for  the  appraisal  team  members. 

Team  members  who  have  previously  been  trained  as  a  member  of  a  prior  appraisal  team  are 
not  automatically  qualified  to  participate  on  a  subsequent  appraisal  without  attending  method 
training.  The  appraisal  team  leader  must  understand  the  nature  of  the  training  delivered 
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previously  and  the  adequacy  of  that  training  for  the  appraisal  at  hand,  which  requires  a 
comparison  between  the  previous  appraisal  and  the  planned  appraisal. 

There  must  be  at  least  one  event  where  the  team  gathers  as  a  group  for  the  purpose  of 
establishing  team  norms  and  operational  decisions  about  how  the  team  will  work  for  the 
appraisal  at  hand. 

Optional  Practices — Some  organizations  have  established  an  “organic”  capability  to 
perform  appraisals  with  very  limited  preparation  through  the  use  of  a  pool  of  trained  Lead 
Assessors  or  appraisal  team  members.  Drawing  from  an  established  group  of  experts  who  are 
accustomed  to  working  together  clearly  provides  a  savings  over  time  for  organizations  that 
conduct  frequent  appraisals. 

Implementation  Guidance — The  team  training  event  is  a  good  place  to  review  the  appraisal 
plan  with  appraisal  team  members.  (Send  the  plan  to  team  members  in  advance.)  This  event 
provides  the  orientation  that  appraisal  team  members  need  to  execute  their  roles 
appropriately,  in  keeping  with  the  “Provide  appraisal  plan  to  relevant  stakeholders  for 
review”  required  practice  in  activity  2. 1.2.6  Obtain  Commitment  to  Appraisal  Plan. 

Additionally,  the  team  training  event  is  a  primary  opportunity  to  conduct  activity  2. 1 .5. 1 
Perform  Readiness  Review.  The  assembled,  trained  appraisal  team  can  then  assess  the 
organization’s  readiness  for  the  appraisal  and  validate  the  reasonableness  of  appraisal 
estimates. 

For  CMMI,  refer  to  the  SEI’s  CMMI  Steward. 

Implementation  Guidance:  Training  in  the  Reference  Model — A  typical  model  training 
course  is  delivered  in  two-and-a-half  to  three  days  for  each  appraisal  method.  The  successful 
completion  of  reference  model  training  should  precede  training  in  the  appraisal  method. 

There  is  no  “aging”  requirement  for  when  model  training  was  received,  but  the  appraisal 
team  leader  is  responsible  for  ensuring  that  each  team  member  has  adequate  reference  model 
understanding  and  for  taking  remedial  action  if  necessary.  Attendance  at  model  training  needs 
to  be  recorded  by  the  training  instructor  and  provided  to  the  People  CMM  Steward,  in 
accordance  with  the  terms  of  instructor  authorization. 

For  appraisals  that  include  higher  levels  (i.e.,  maturity  levels  4  and  5)  team  members  might 
benefit  from  additional  training,  such  as  attending  a  course  on  statistical  process  control  or 
other  advanced  topics. 

Implementation  Guidance:  Training  in  the  Appraisal  Method — A  typical  delivery  of 
appraisal  team  training  might  take  two-and-a-half  to  three  days.  More  or  less  time  might  be 
necessary  depending  on  the  experience  of  the  appraisal  team  members. 

Exercises  in  appraisal  techniques  and  team  development  are  used  to  reinforce  the  skills  that 
are  important  during  the  appraisal.  Exercises  should  be  used  that  are  appropriate  for  the 
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organization  being  appraised.  Where  sufficient  organizational  artifacts  exist,  “live”  data  can 
be  collected  and  used  in  training  exercises,  where  appropriate.  Just-in-time  training  can  also 
be  used  to  re-emphasize  method  concepts  at  the  points  in  the  appraisal  process  when  the 
skills  are  to  be  used. 

Appraisal  team  training  materials  should  be  tailored  to  fit  team  needs  and  objectives  for  the 
specific  appraisal.  Tailoring  provides  opportunities  to 

•  provide  insight  into  the  context,  objectives,  and  plans  of  the  particular  appraisal 

•  communicate  team  members’  assigned  roles  and  responsibilities 

•  identify  tailoring  of  SCAMPI  with  People  CMM  for  the  upcoming  appraisal 

•  acquaint  the  team  with  the  appraised  organization’s  characteristics  and  documentation 

•  focus  on  skills  that  might  be  critical  to  the  upcoming  appraisal,  such  as  the  ability  to 
facilitate  interviews  or  the  ability  to  identify  alternative  practices 

This  training  should  be  provided  within  60  days  of  the  appraisal.  The  appraisal  team  leader 
typically  provides  method  training,  but  other  delivery  options  are  also  acceptable  (as 
described  above).  Although  alternative  training  options  can  provide  some  advantages  and 
efficiencies  for  method  training,  there  are  also  potential  consequences  that  might  be  felt  by 
the  appraisal  team  leader  on  a  given  appraisal,  such  as  poor  training  quality  or  readiness  of 
team  members.  Regardless  of  how  method  training  is  delivered  to  the  team  members, 
opportunities  for  team  building  should  be  provided  to  unite  the  team  and  bring  them  up  to 
speed  on  the  specifics  of  the  planned  appraisal. 

Implementation  Guidance:  Familiarization  with  the  Appraisal  Plan — Method  training 
and  team  building  provide  good  opportunities  to  familiarize  the  team  with  the  appraisal  plan, 
including  appraisal  objectives,  organizational  scope,  reference  model  scope,  and  the 
schedule,  resources,  and  constraints.  Team  member  input  can  be  obtained  to  refine  or 
complete  the  contents  of  the  appraisal  plan. 

Implementation  Guidance:  Analysis  of  Objective  Evidence — Analysis  of  the  objective 
evidence  provided  by  the  appraised  organization,  such  as  questionnaire  responses  or 
worksheets  summarizing  objective  evidence,  can  be  accomplished  as  an  integrated  part  of 
appraisal  team  preparation  and  training,  or  afterwards. 

Team  members  should  become  familiar  with  the  instruments  (e.g.,  document  inventory  list, 
questionnaires,  PII  database)  to  be  used  as  data  collection  sources  during  the  appraisal. 
Demonstrations  or  exercises  using  the  data  collection  tools  and  methods  planned  for  the 
appraisal  should  provide  appraisal  team  members  with  an  opportunity  to  practice  techniques 
for  data  recording,  verification,  and  analysis.  This  might  include  mechanisms  such  as  wall 
charts,  spreadsheets,  or  data  reduction  tools.  The  more  familiarity  and  comfort  developed 
using  these  tools,  the  greater  the  savings  in  team  efficiency  during  the  appraisal  on-site 
phases. 
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Note  that  this  interpretation  guide  only  addresses  Class  A  SCAMPI  with  People  CMM 

appraisals,  where  PII  databases  are  required. 

Implementation  Guidance:  Roles  and  Responsibilities — The  appraisal  team  leader  should 

assign  and  explain  team  member  roles  and  responsibilities  for  the  appraisal.  Typical  roles  to 

be  assigned  include: 

•  appraised  organization  coordinator — handles  on-site  logistics  and  provides  technical, 
administrative,  and  logistical  support  to  the  appraisal  team  leader.  This  usually  includes 
coordinating  schedules,  notifying  participants,  arranging  adequate  facilities  and 
resources,  obtaining  requested  documentation,  and  arranging  catering.  He  or  she  might 
also  coordinate  or  provide  clerical  support  to  the  team.  This  role  is  often  assigned  to  one 
or  more  members  of  the  appraised  organization.  The  appraised  organization  coordinator 
might  be  one  of  the  appraisal  team  members,  or  the  role  could  be  assigned  to  other  site 
personnel. 

•  librarian — manages  the  inventory  of  appraisal  documents,  coordinates  requests  for 
additional  documentation,  and  returns  documents  at  the  end  of  the  appraisal.  This  role 
can  be  fulfilled  by  an  appraisal  team  member  or  by  a  member  of  the  support  staff. 

•  process  area  mini-teams — take  the  lead  in  data  collection  in  assigned  PAs.  They  ensure 
that  information  collected  during  a  data  gathering  session  covers  their  PAs,  request 
additional  information  needed  relative  to  their  PAs,  and  record  the  work  performed  by 
individual  appraisal  team  members  pertaining  to  their  PAs. 

mini-teams  typically  consist  of  two  or  three  members.  Mini-team  assignments  can  be 
made  based  on  several  factors,  including  the  following: 

-  related  PAs  (e.g.,  PA  categories) 

-  the  mix  of  experience  of  mini-team  members  (e.g.,  discipline  experience,  appraisal 
experience) 

•  facilitator — conducts  interviews  by  questioning  interview  participants.  A  facilitator  is 
available  for  each  parallel  interview  session,  and  the  Lead  Appraiser  moves  between  the 
parallel  sessions  to  verify  the  process  performance  of  the  interviews. 

•  timekeeper— responsible  for  tracking  time  and  schedule  constraints  during  interviews 
and  other  activities. 

•  observer — due  to  the  confidentiality  required  during  an  appraisal  and  the  cohesiveness 
needed  to  participate  in  appraisal  activities,  observers  are  not  permitted  to  participate  in 
the  appraisal  processes.  The  only  exception  is  an  observer  who  is  authorized  by  the 
People  CMM  Steward  to  observe  a  candidate  Lead  Appraiser’s  performance  as  appraisal 
team  leader  or  to  perform  an  audit  as  part  of  the  quality  audit  function  of  the  steward. 

Note  that  some  appraisal  team  members  are  not  trained  to  participate  with  both  models 
and  methods  when  a  combined  SCAMPI  with  People  CMM  and  SCAMPI  appraisal  is 
performed.  In  these  cases,  the  appraisal  team  members  can  sit  with  the  team  throughout 
the  consensus,  but  cannot  participate  in  the  consensus  discussions  for  the  models  in 
which  they  are  not  trained  and  qualified. 
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2.1 .4  Obtain  and  Analyze  Initial  Objective  Evidence 

Purpose— Obtain  information  that  facilitates  site-specific  preparation  and  an  understanding 
of  the  implementation  of  model  practices  across  the  appraised  organization.  Identify  potential 
issues,  gaps,  or  risks  to  aid  in  refining  the  plan.  Strengthen  understanding  of  the 
organization’s  operations  and  processes. 

Entry  Criteria 

•  appraisal  input  received 

•  sponsor  authorization  to  proceed 

•  availability  of  practice  implementation  data  for  appraised  organization 

Inputs 

•  practice  implementation  data  for  appraised  organization 

•  identified  participants 

Activities — The  six  activities  listed  below  are  required  for  this  process. 

2. 1.4.1  Plan  Survey  Administration 

2. 1.4.2  Prepare  Participants 

2. 1.4.3  Administer  Instruments 

2. 1.4.4  Analyze  People  CMM  Survey  Results 

2. 1.4.5  Obtain  Initial  Objective  Evidence 

2. 1 .4.6  Inventory  Objective  Evidence 

Outputs 

•  completed  instruments 

•  data  analyses  results  (e.g.,  data  summaries,  questionnaire  results) 

•  identification  of  additional  information  needed 

•  prepared  participants 

•  initial  set  of  objective  evidence 

Outcome 

•  Initial  objective  evidence  has  been  collected,  organized,  and  recorded. 

•  Potentially  important  areas  of  needed  information  have  been  noted. 

•  The  team  has  a  deeper  understanding  of  the  appraised  organization’s  operations  and 
processes. 

•  The  team  is  ready  to  make  detailed  plans  for  data  collection. 

Exit  Criteria 

•  All  objective  evidence  captured  during  this  activity  has  been  recorded  for  later  use. 
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•  High-priority  areas  for  additional  data  collection  have  been  identified. 

•  The  level  of  sufficiency  of  the  objective  evidence  to  support  the  appraisal  is  determined. 

Key  Points— Gather  high-leverage  objective  evidence.  The  amount  of  initial  objective 
evidence  provided  by  the  organization  will  determine  the  proportion  of  evidence  that  must  be 
discovered  (versus  verified)  during  the  appraisal.  Maximizing  time  spent  in  verification 
(versus  discovery)  is  a  key  performance  objective  for  the  appraisal  process. 

Tools  and  Techniques 

•  Automated  support  for  questionnaires,  including  data  reduction  tools,  might  be  available 
to  make  data  analysis  more  efficient. 

•  Breaking  into  mini-teams  to  review  data  related  to  specific  PAs  is  a  way  to  ensure 
completeness  in  the  data. 

Metrics 

•  number  of  practices  for  which  complete  objective  evidence  is  available 

•  number  of  questionnaire  respondents  reported  in  the  appraisal  record 

•  calendar  time  and  effort  expended  for  this  activity  compared  to  the  planned  values 

Metrics  are  captured  in  the  appraisal  plan. 

Verification  and  Validation— Where  the  team  includes  members  of  the  appraised 
organization,  these  members  should  be  used  to  help  understand  the  initial  objective  evidence 
provided  to  prevent  misinterpretation  of  terms  or  special  conditions. 

Inconsistencies  and  contradictions  among  the  items  provided  in  initial  objective  evidence 
should  be  identified  and  recorded  for  resolution. 

Records— Records  of  this  process  include  completed  and/or  summarized  questionnaires, 
profiles,  and  surveys.  Lists  of  information  needed  should  be  maintained  and  used  as  an  input 
to  the  later  data  collection  activities.  Calendar  time  and  effort  expended  in  this  activity  should 
be  recorded  and  compared  to  the  plan.  These  data  are  part  of  the  appraisal  record. 

Tailoring — A  variety  of  methods  can  be  used  to  collect  initial  data,  including  the  following: 

•  a  site  information  package  prepared  by  representatives  of  the  organization 

•  a  presentation  on  the  process  improvement  program  and  its  accomplishments 

•  specialized  or  general  questionnaires  focused  on  practice  implementation 

The  use  of  additional  instruments  is  dependent  on  the  results  of  the  analysis  of  available  data 
and  the  results  of  process  2.1.5  Prepare  for  Collection  of  Objective  Evidence. 

Interfaces  with  Other  Processes — This  process  plays  a  critical  role  in  the  planning  and 
preparation  processes.  The  information  generated  in  this  process  provides  the  most  important 
opportunity  to  reset  expectations  and  plans  with  the  appraisal  sponsor  if  initial  assumptions 
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about  the  availability  of  objective  evidence  are  incorrect.  It  also  provides  the  basis  of  data 
collection  planning. 

Summary  of  Activities — The  appraisal  team  leader  works  with  representatives  of  the 
organization  to  obtain  an  initial  data  set  that  represents  an  inventory  of  the  objective  evidence 
pertaining  to  the  implementation  of  each  instantiation  of  each  practice  within  the  appraisal 
scope.  This  initial  data  set  is  first  reviewed  by  the  appraisal  team  leader  for  a  high-level 
assessment  of  adequacy  and  completeness.  The  appraisal  team  leader  or  appraisal  team  then 
performs  a  more  detailed  analysis  to  use  as  input  for  planning  data  collection  and  verification 
activities  that  occur  on  site.  Finally,  a  record  is  created  with  a  detailed  accounting  of  any 
missing  objective  evidence.  This  record  is  used  as  primary  input  for  the  data  collection  plan. 

2.1 .4.1  Plan  Survey  Administration 

Activity  Description — The  surveying  phase  of  a  People  CMM  appraisal  involves  all  aspects 
of  collecting  and  analyzing  data  from  a  People  CMM  survey.  The  purpose  of  the  People 
CMM  survey  is  to  collect  information  about  workforce  practices  from  a  broad  sample  of 
people  working  in  the  organization. 

Activities  performed  during  this  task  complete  all  preparations  for  administering  the  People 
CMM  questionnaires.  The  Lead  Appraiser  and  site  coordinator  should  plan  all  aspects  of 
administering  the  People  CMM  questionnaires.  All  necessary  arrangements  should  be  made 
for  administering  the  questionnaires,  including  a  specification  for  the  sample  of  people  to 
whom  People  CMM  questionnaires  will  be  administered.  Samples  of  individuals  in  the 
organization  are  selected  to  complete  a  People  CMM  questionnaire. 

The  People  CMM  maturity  questionnaires  are  designed  to  collect  data  regarding  practices  in 
each  key  process  area  of  the  People  CMM.  There  are  two  different  questionnaires 
administered  during  this  phase:  one  for  managers/supervisors  and  one  for  non¬ 
managers/individual  contributors. 

In  addition  to  planning  the  logistics,  the  language  in  each  question  should  be  reviewed  in 
light  of  local  terms  or  jargon  to  identify  any  terminology  that  is  likely  to  be  misinterpreted. 

Required  Practices — Some  of  the  activities  typically  performed  during  this  task  are  listed 
below. 

•  Identify  sample  characteristics. 

•  Identify  the  number  of  people  to  be  included  in  the  sample. 

•  Select  participants  to  complete  People  CMM  questionnaires. 

•  Ensure  adequate  survey  coverage  across  the  organization. 

•  Plan  survey  administration  and  logistics. 

•  Arrange  final  questionnaire  administration  logistics. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 
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Parameters  and  Limits— In  order  to  select  a  representative  cross  section  of  survey 
respondents,  the  following  guidelines  should  be  applied  in  randomly  selecting  individuals  to 
respond  to  the  surveys.  Sampling  guidelines  based  on  organization  size  are  provided  below. 

•  For  organizations  of  50  or  fewer  people,  everyone  should  complete  a  questionnaire. 

•  For  organizations  of  51-200  people,  questionnaires  should  be  given  to  at  least  a  50% 
sample,  with  a  minimum  of  at  least  50  people  completing  questionnaires. 

•  For  organizations  of  more  than  200  people,  questionnaires  should  be  given  to  at  least  a 
20%  sample,  with  a  minimum  of  at  least  100  people  completing  questionnaires. 

Sampling  guidelines  based  on  job  assignment  are  provided  below. 

•  Approximately  25%  -  50%  (and  possibly  up  to  100%  for  small  organizations)  of  the 
managers  ranging  from  first-line  supervisors  through  executives  should  complete 
questionnaires. 

•  At  least  20%  of  the  non-managers  spanning  the  entire  scope  of  the  organization  should 
complete  questionnaires,  with  individuals  being  selected  to  ensure  an  unbiased  sampling 
with  respect  to  department,  division,  or  other  major  organizational  component,  such  as 

-  assigned  workgroup 

-  grade  or  level 

-  job  type  or  category 

-  type  of  work  or  project 

-  other  important  characteristics  over  which  experiences  or  perceptions  of  workforce 
practices  might  differ,  such  as  gender  or  length  of  service  with  the  assessed 
organization 

Refer  to  Appendix  A  for  information  about  how  to  sample. 

Optional  Practices— Identify  questionnaire  terminology  that  is  likely  to  be  misinterpreted. 

Implementation  Guidance— This  sample  should  be  carefully  planned  to  ensure  adequate 
balance  and  coverage  across  types  of  jobs  and  appraised  organizations.  The  questionnaires 
should  be  administered  to  a  mix  of  employees  proportional  to  the  organization  s  population 
as  a  whole. 

2.1 .4.2  Prepare  Participants 

Activity  Description — Members  of  the  organization  who  participate  in  the  appraisal  must  be 
informed  of  their  role  and  the  expectations  of  the  sponsor  and  appraisal  team.  This  is 
typically  accomplished  through  a  briefing  where  the  appraisal  team  leader  provides  an 
overview  of  the  appraisal  process,  purpose,  and  objectives.  Specific  information  about  the 
scheduled  events  and  locations  is  also  communicated  during  this  presentation,  as  well  as 
through  ongoing  contact  between  the  appraised  organization  coordinator  and  the  members  of 
the  organization. 
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Required  Practices  for  Participants  to  be  Interviewed 

•  Send  an  invitation  to  participate  to  the  survey  sample. 

•  Provide  confirmation  of  the  survey  schedule  and  logistics  to  participants. 

•  Brief  appraisal  participants  on  the  appraisal  process. 

•  Provide  orientation  to  appraisal  participants  on  their  roles  in  the  appraisal. 

Required  Practices  for  Appraisal  Team  Participants 

•  Review  scheduled  activities,  roles,  logistics,  and  coordination  needs  and  make  necessary 
adjustments. 

•  Sign  confidentiality  agreement. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — The  orientation  provided  to  appraisal  participants  must  occur 
some  time  prior  to  their  participation  so  they  can  confirm  their  availability  and  prepare  for 
their  participation.  The  preparation  of  appraisal  participants  may  be  accomplished  by  video  or 
teleconference  if  desired. 

Optional  Practices — Provide  orientation  on  the  documentation  of  PIIs  (see  2. 1.4.3 
Administer  Instruments  and  Appendix  G)  and  any  specific  instruments  used,  so  the 
appropriate  people  in  the  organization  can  document  the  initial  objective  evidence  to  be  used 
in  the  appraisal. 

Implementation  Guidance — The  appraisal  team  leader  will  likely  work  with  senior 
management  to  help  demonstrate  the  sponsor’s  commitment  to  the  appraisal  process  and  the 
process  improvement  work  that  will  follow.  However,  in  large  organizations,  the  possibility 
of  the  same  team  visiting  multiple  sites  adds  coordination  tasks  and  communication  channels 
as  well. 

Appraisal  participants  should  also  be  informed  of  the  need  to  provide  accurate  and  complete 
information  on  instruments.  In  addition  to  assisting  with  appraisal  accuracy,  this  can  help  to 
ensure  sufficient  coverage  of  reference  model  practices  and  reduce  the  amount  of  time 
necessary  for  follow-up  interviews.  The  investment  in  initial  population  of  complete 
"instruments,  such  as  PIIs  (see  2. 1.4.3  Administer  Instruments  and  Appendix  G), 
questionnaires,  or  mapping  tables  can  be  recovered  by  reduced  effort  in  the  reuse  of  assets  for 
subsequent  appraisals. 

Activities  performed  during  this  task  orient  all  appraisal  participants  to  the  appraisal  process 
and  their  role  in  it.  This  meeting  is  used  to  begin  the  on-site  appraisal  activities  with 
participants  and  to  remind  participants  of  the  schedule  and  location  of  events  in  which  they 
are  involved.  All  appraisal  participants  should  attend  this  meeting,  including 

•  appraisal  team  members 

•  sponsor 
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•  all  people  scheduled  to  participate  in  questionnaires,  interviews  or  discussions 

•  others  (occasionally  customers)  at  the  discretion  of  the  sponsor 

In  addition  to  a  presentation  by  the  Lead  Appraiser,  the  appraisal  sponsor  and  site  coordinator  may 
also  give  presentations.  Some  of  the  topics  typically  presented  include  the  following: 

•  appraisal  objectives 

•  introduction  of  appraisal  team  members 

•  appraisal  principles  (especially  confidentiality) 

•  appraisal  activities  conducted  to  date 

•  appraisal  process  flow 

•  times  and  locations  of  activities  involving  participants  (including  changes) 

•  relevant  logistics 

Activities  performed  during  this  task  allow  the  appraisal  team  time  to  review  assignments 
and  schedules  and  complete  any  preparations  for  the  appraisal  prior  to  the  opening  briefing 
for  participants.  During  this  meeting,  all  members  of  the  appraisal  team  sign  the 
confidentiality  agreement.  This  agreement  is  provided  to  each  on-site  appraisal  participant, 
and  they  are  asked  to  sign  it  (although  some  organizations  choose  not  to  use  this  agreement 
with  participants). 

2.1 .4.3  Administer  Instruments 

Activity  Description — This  activity  involves  the  administration  of  instruments  for  the 
appraisal  in  addition  to  the  input  data,  such  as  process  implementation  indicators  provided  by 
the  organization  as  input  to  the  appraisal.  It  includes  the  use  of  structured  techniques  and 
work  aids  (e.g.,  surveys,  questionnaires,  or  an  objective  evidence  database  [see  Appendix  G]) 
to  assist  the  appraised  organization  in  characterizing  their  process  and  supporting  objective 
evidence  in  terms  of  model  practices. 

A  practice-based  questionnaire  is  also  a  commonly  used  instrument.  Such  questionnaires 
typically  have  a  series  of  focused  questions,  each  one  providing  an  opportunity  for  the 
respondent  to  answer  a  closed-ended  question  about  a  practice.  In  addition,  the  respondent  is 
given  an  opportunity  to  write  a  clarifying  comment  that  elaborates  on  the  closed-ended 
response. 

Activities  performed  during  this  task  result  in  the  collection  of  data  from  the  People  CMM 
survey  and  the  preparation  of  the  data  for  analysis.  The  survey  participants  attend  a  survey 
administration  session  where  they  are  given  a  briefing  that  explains  the  People  CMM  and  its 
appraisal  process.  The  number  of  survey  administration  sessions  depends  on  the  number  of 
'survey  participants  and  the  size  of  facilities  available  for  this  session.  The  session  is 
conducted  by  the  Lead  Appraiser  or  designee,  together  with  an  appraisal  team  member  from 
the  site,  typically  the  site  coordinator.  The  Lead  Appraiser  or  designee  presents  a  short 
briefing  describing  the  People  CMM,  the  purpose  of  the  survey,  and  its  role  in  the  appraisal 
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process.  During  this  session  the  questionnaires  or  access  to  electronic  versions  are  given  to 
respondents.  Several  options  might  be  offered  for  completing  the  survey,  including  the 
following: 

•  in  a  group  session 

•  individually  outside  of  the  group  session 

•  online 

Unless  participants  can  complete  their  questionnaires  online,  it  is  best  for  them  to  complete 
questionnaires  in  a  group  session  because  appraisal  team  members  will  be  available  to  help 
them  understand  the  intent  of  the  questions  and  provide  directions  for  responding.  Certain 
questions,  such  as  questions  regarding  location  or  employer,  might  require  instructions  for  the 
participants  that  are  unique  to  the  appraisal. 

Questionnaires  are  collected  and  sent  for  scoring.  Questionnaires  and  scoring  services  are 
available  as  part  of  the  SEI-authorized  appraisal  kits.  Responses  to  the  People  CMM  survey 
are  scored  and  prepared  into  summary  reports  that  describe  the  results  for  both  the  individual 
and  the  manager  questionnaires.  These  reports  are  distributed  to  the  appraisal  team  prior  to 
the  on-site  appraisal. 

Required  Practices — Consider  using  the  People  CMM  survey  instrument.  While  the  People 
CMM  survey  is  optional  for  level  4  and  5  organizations,  it  is  recommended  for  lower 
maturity  organizations. 

Some  of  the  activities  typically  performed  during  this  task  are  listed  below. 

•  Brief  People  CMM  survey  participants  on  the  People  CMM,  the  guidelines  for 
questionnaire  completion,  and  the  overall  appraisal  process. 

•  Monitor  the  completion  of  surveys  and  provide  necessary  interpretation  and  guidance  to 
participants. 

•  Score  survey  responses  and  distribute  results  to  the  appraisal  team  members. 

•  Administer  planned  appraisal  instruments  for  the  entry  of  data  by  appraisal  participants. 

Parameters  and  Limits —  Only  the  instruments  identified  in  the  data  collection  plan  can  be 
used.  Instruments  are  typically  administered  by  representatives  of  the  appraisal  team,  with  the 
appraisal  team  leader  responsible  for  negotiating  additional  time  and  resources  if  the  data  is 
incomplete.  The  appraisal  team  leader  is  also  responsible  for  making  sure  duplicate  data  entry 
on  multiple  instruments  is  not  requested.  No  organization  should  be  asked  to  provide  the 
same  information  in  two  or  more  formats. 

Whatever  vehicle  is  used,  the  resultant  data  must  provide  information  about  the  extent  of  the 
implementation  of  model  practices  in  the  appraised  organization  and  the  sampled  units. 

Optional  Practices — Establish  an  organizational  asset  (or  rely  on  an  existing  one)  that 
documents  and  maintains  the  traceability  of  implemented  practices  to  model  practices. 
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The  ATL  should  consider  conducting  a  workshop  to  document  the  PIIs  for  the  organization, 
especially  when  the  organization  is  new  to  SCAMPI  with  People  CMM.  The  workshop  will 
improve  understanding  of  the  ways  in  which  PIIs  are  to  be  used  and  instantiated  by  the  units 
included  in  the  appraisal. 

Implementation  Guidance — Using  instruments  to  gather  written  information  from  members 
of  the  organization  provides  a  relatively  low-cost  data  collection  technique  when  done  well. 
This  type  of  data  tends  to  be  most  useful  when  provided  early  in  the  appraisal  conduct  and 
can  lead  to  valuable  insights  about  where  data  might  be  sought  during  subsequent  data 
collection  events.  Since  there  is  limited  opportunity  for  elaboration  and  branching  to  related 
topics,  responses  to  instruments  can  sometimes  raise  more  questions  than  they  answer. 
Furthermore,  instruments  that  contain  excessive  jargon  or  complicated  terminology  might 
hinder  data  collection.  Confused  respondents  will  do  their  best  to  answer  the  question  they  do 
not  quite  understand,  and  the  responses  will  be  interpreted  based  on  the  question  that  was 
intended.  Having  a  knowledgeable  person  present  during  the  administration  of  an  instrument 
can  help  prevent  miscommunication. 

Instruments  are  attractive  for  data  collection  because  they  can  be  used  to  establish  a  “scoring 
scheme”  that  reduces  the  burden  of  interpretation  for  the  data  recipient.  Such  schemes  do  not 
exist  for  SCAMPI  with  People  CMM,  and  the  use  of  a  shortcut  of  this  type  is  a  violation  of 
the  principle  that  focuses  rating  judgments  on  the  goals  of  the  PAs  in  People  CMM  models. 
The  practices  found  in  the  People  CMM  model  are  expected  components,  while  the  goals  in 
the  model  are  required  components.  While  the  satisfaction  of  a  PA  goal  is  predicated  on  the 
implementation  of  practices  found  in  the  model  (or  acceptable  alternatives),  there  is  no  strict 
aggregation  scheme  that  allows  the  inference  of  goal  satisfaction  based  on  practice 
implementation  alone.  Rating  judgments  are  based  on  multiple  sources  of  objective  evidence 
and  the  reasoned  consideration  of  strengths  and  weaknesses,  in  aggregate. 

Whenever  possible,  documents  mentioned  in  the  responses  to  questionnaires  or  other 
instruments  should  be  requested  for  team  review  early  in  the  process  so  any  misleading 
references  will  not  cause  undue  confusion  later. 

2.1 .4.4  Analyze  People  CMM  Survey  Results 

Activity  Description — Activities  performed  during  this  task  provide  information  about  the 
consistency  of  the  performance  of  workforce  practices  and  the  major  issues  related  to  them. 

Survey  results  are  provided  to  each  member  of  the  appraisal  team,  including  the  responses  to 
the  surveys  completed  by  managers  and  non-managers.  For  each  individual  question,  the 
report  provides  summary  statistical  data  and  written  comments  related  to  that  question.  Each 
member  of  the  appraisal  team  should  review  all  responses  in  the  survey  report  to  develop  an 
initial  impression  of  the  workforce  issues  the  organization  is  facing. 

Results  from  analyzing  survey  data  are  used  in  developing  the  interview  scripts  employed 
during  the  on-site  appraisal.  No  rating  decisions  are  made  based  solely  on  survey  responses. 
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Required  Practices — Some  of  the  activities  performed  during  this  task  include  the 
following: 

•  review  the  survey  results 

•  reach  consensus  on  interpretation  of  the  survey  data 

•  consolidate  survey  data  on  process  area  worksheets 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — The  survey  results  must  be  reviewed  by  the  assigned  appraisal 
team  members  for  their  respective  PAs.  The  mini-team  partners  should  also  review  the  survey 
results  for  their  respective  mini-team  partner’s  assigned  PAs.  The  mini-team  should  reach 
consensus  on  the  interpretation  of  the  results.  Any  disputes  or  problems  with  interpretation 
should  be  reviewed  with  the  ATL.  When  the  full  team  performs  its  first  consensus  review, 
which  will  most  likely  include  documents  and  some  interview  findings,  particular  attention 
should  be  given  to  the  survey  data  findings. 

Survey  data  must  be  entered  into  the  process  area  workbook  (PAWB)  worksheets  (see 
Appendix  J)  when  the  survey  is  used  as  an  instrument  to  obtain  information  about  the 
appraised  organization. 

Optional  Practices — None. 

Implementation  Guidance — The  SCAMPI  with  People  CMM  appraisal  process  can  be 
organized  to  incorporate  this  task  into  the  assessing  phase.  This  task  can  be  completed  before 
the  on-site  appraisal,  as  depicted  here,  or  as  a  task  included  during  the  on-site  activities. 

2.1. 4.5  Obtain  Initial  Objective  Evidence 

Activity  Description — The  appraisal  team  leader  asks  the  organization  to  provide  detailed 
data  on  the  implementation  of  practices  in  the  organization.  The  appraisal  team  leader  can 
specify  the  format  to  be  used  and  the  level  of  detail  to  be  provided,  keeping  in  mind  that 
anything  not  provided  in  advance  must  be  collected  later  in  the  appraisal  process.  There  are 
no  minimum  requirements  set  by  the  method  for  the  completeness  or  detail  of  the  initial  data 
set.  However,  the  effort  required  to  conduct  a  SCAMPI  with  People  CMM  appraisal  is  a 
direct  function  of  the  amount  of  data  available  to  the  team  at  the  beginning  of  the  process. 
Before  the  appraisal  outputs  can  be  created,  the  team  needs  to  verify  objective  evidence  for 
each  instantiation  in  each  unit  of  each  practice  within  the  scope  of  the  appraisal.  For  detailed 
requirements  on  data  sufficiency,  refer  to  process  2.2.2  Verify  and  Validate  Objective 
Evidence. 

The  use  of  a  completely  populated  PII  database  (i.e.,  all  requested  artifacts  have  been 
provided  to  the  appraisal  team  prior  to  the  document  reviews  before  the  interviews)  is 
desirable  but  not  essential  at  this  stage  in  the  appraisal  process.  The  appraisal  team  leader 
must  give  the  organization  an  opportunity  to  provide  it,  but  will  not  require  it  unless  the 
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sponsor  has  agreed  that  this  will  be  a  verification-oriented  appraisal  (i.e.,  information  is 
obtained  during  the  in  interview  cycles,  as  opposed  to  a  discovery-oriented  appraisal). 

A  “mapping”  of  implemented  practices  and  model  practices  is  required,  and  might  be 
generated  using  questionnaires  (see  activity  2. 1.4. 3  Administer  Instruments). 

Required  Practices — Obtain  documentation  reflecting  the  implementation  of  model 
practices  within  the  appraised  organization  and  sampled  units. 

Parameters  and  Limits — At  a  minimum,  the  organization  must  provide  a  list  of  documents 
that  are  relevant  to  understanding  the  processes  in  use  in  the  appraised  organization  and  the 
sampled  units.  This  list  must  be  mapped  to  the  model  practices  that  are  in  the  scope  of  the 
appraisal. 

Optional  Practices — A  list  of  terms  and  important  jargon  used  in  the  appraised  organization 
might  be  provided  to  the  team  to  aid  communication  with  the  members  of  the  organization. 

A  complete  objective  evidence  database,  which  documents  the  implementation  of  every 
model  practice  within  the  scope  of  the  appraisal  in  the  appraised  organization  and  the 
sampled  units  may  be  provided  to  the  team  in  advance.  The  use  of  database  tools  specifically 
built  to  support  a  process  appraisal  is  highly  recommended. 

Implementation  Guidance — Whether  collected  through  instruments,  the  review  of 
documents,  attending  presentations,  or  interviews,  the  data  used  for  an  appraisal  is  related  to 
the  practices  of  the  reference  model.  For  every  practice  within  the  model  scope  of  the 
appraisal  and  for  every  instance  of  each  practice,  objective  evidence  is  used  as  the  basis  for 
appraisal  team  determinations  of  the  extent  to  which  the  practice  is  implemented.  Indicators 
that  substantiate  practice  implementation  include  the  following: 

•  direct  artifacts,  which  represent  the  primary  tangible  output  of  a  practice.  These  are 
sometimes  listed  in  the  People  CMM  model  as  examples  or  can  be  derived  from  the 
practice  and  sub-practice  descriptions  One  or  more  direct  artifacts  might  be  necessary  to 
verify  the  implementation  of  associated  model  practices. 

•  indirect  artifacts,  which  represent  artifacts  that  are  a  consequence  of  performing  the 
practice  but  not  necessarily  the  purpose  for  which  it  was  performed.  These  are  typically 
things  like  meeting  minutes,  review  results,  or  written  communications  of  status. 

•  affirmations,  which  are  oral  or  written  statements  confirming  the  implementation  of  the 
practice.  These  are  typically  validated  using  interviews,  questionnaires,  or  other  means 
during  the  onsite  period. 

Not  all  practices  have  indirect  artifacts.  (See  Appendix  B.) 

Prior  to  the  data  collection  activities  carried  out  by  the  appraisal  team,  an  initial  data  set  is 
usually  created  by  the  appraised  organization.  This  data  set  contains  descriptions  of  the 
objective  evidence  available  for  the  team  to  examine,  complete  with  references  to 
documentation  and  identification  of  the  personnel  who  can  provide  relevant  affirmations. 
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This  instrument  provides  the  baseline  of  objective  evidence  for  the  appraisal.  Most 
organizations  with  experience  in  process  improvement  will  already  have  this  type  of  data  on 
hand  since  they  will  have  used  it  to  track  their  progress. 

Artifacts  might  be  obtained  as  hard  copies,  soft  copies,  or  hyperlinks  to  the  documents  in  a 
Web-based  environment.  If  hyperlinks  are  used,  the  accessibility  of  artifacts  using  these  links 
should  be  verified  in  the  appraisal  environment.  For  example,  appraisal  team  access  could  be 
inhibited  by  invalid  references  or  firewalls. 

The  initial  data  set  forms  the  basis  for  planning  the  data  collection  activities,  including 
interviews  and  presentations  on  site.  Any  objective  evidence  that  is  not  identified  before  the 
team’s  arrival  will  need  to  be  sought  by  the  team  members  once  they  arrive  on  site.  This 
process  of  “discovering”  whether — and  how — the  organization  has  addressed  a  given 
practice  in  the  model  can  be  quite  time  consuming,  and  it  is  often  difficult  to  predict  how 
long  it  will  take. 

2.1 .4.6  Inventory  Objective  Evidence 

Activity  Description — The  analysis  of  the  initial  data  set  provides  critical  new  information 
for  the  overall  planning  of  the  appraisal  and  forms  the  basis  for  the  detailed  data  collection 
plan  that  must  be  developed  before  the  on-site  data  collection  begins.  The  analysis  of  initial 
objective  evidence  at  this  stage  is  focused  primarily  on  the  adequacy  and  completeness  of 
information  and  the  implications  for  future  data  collection.  The  results  of  this  analysis  are  the 
primary  basis  for  determining  the  extent  to  which  the  appraisal  will  be  one  of  verification  or 
discovery. 

Required  Practices 

•  Examine  the  initial  set  of  objective  evidence  provided  by  the  appraised  organization. 

•  Determine  the  extent  to  which  additional  information  is  needed  for  adequate  coverage  of 
model  practices. 

Parameters  and  Limits — Information  provided  by  the  appraised  organization  must  be 
detailed  enough  to  show  the  extent  to  which  each  type  of  objective  evidence  (direct  artifacts, 
indirect  artifacts,  and  affirmations)  is  available  for  each  process  instantiation  for  each  model 
practice  within  the  scope  of  the  appraisal.  This  initial  review  of  objective  evidence  identifies 
model  practices  for  which  the  team  has 

•  strong  objective  evidence 

•  no  objective  evidence 

•  conflicting  objective  evidence 

•  anomalous  objective  evidence 

•  insufficient  objective  evidence 
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Key  documents  are  identified  that  can  be  used  to  gain  insight  into  several  model  practices. 
These  are  potential  high-leverage  documents  that  might  be  good  candidates  for  pre-on-site 
review  by  team  members. 

Optional  Practices— Review  the  initial  objective  evidence  with  the  process  owners,  if 
needed,  to  increase  understanding. 

Implementation  Guidance — Members  of  the  team  might  choose  to  summarize  the  extent  of 
practice  implementation  at  the  discretion  of  the  appraisal  team  leader.  However,  the  objective 
of  this  activity  is  to  determine  how  much  additional  data  team  members  will  need  to 
complete  their  work.  It  is  recommended  that  the  appraisal  team  leader  establish  an 
expectation  with  the  sponsor  that  the  results  of  this  analysis  will  form  the  basis  for  a  revised 
schedule  estimate.  If  the  initial  objective  evidence  is  lacking  in  completeness  and  detail,  the 
team  will  be  forced  to  seek  more  information  during  the  on-site  data  collection  unless 
corrective  actions  are  taken  before  that  time. 

It  is  important  to  keep  all  stakeholders  focused  on  the  fact  that  the  SCAMPI  with  People 
CMM  is  intended  as  a  benchmarking  Class  A  appraisal.  This  method  is  not  well  suited  for 
organizations  that  have  very  limited  understanding  of  People  CMM.  Other  organizations 
might  better  benefit  from  a  Class  B  or  C  SCAMPI  with  People  CMM  appraisal,  which  are  yet 
to  be  defined. 

Deciding  on  a  reasonable  implementation  of  the  practices  and  working  to  ensure  that  they  are 
enacted  on  units  throughout  the  organization  are  activities  that  precede  a  benchmarking 
appraisal.  A  different  type  of  appraisal  (Class  B  or  C)  is  likely  more  valuable  if  the  objective 
of  the  sponsor  is  to  begin  the  process  of  understanding  what  People  CMM  could  mean  for  the 
organization.  It  is  not  reasonable  to  schedule  an  appraisal  and  expect  to  collect  all  of  the  data 
required  for  benchmarking  during  the  on-site  data  collection. 

The  appraisal  team  leader  often  reviews  the  initial  data  set  from  the  organization  before 
assembling  the  team  for  its  first  meeting  to  identify  areas  where  additional  data  is  needed  and 
to  assess  the  feasibility  of  the  planned  appraisal  schedule.  This  readiness  review  should  be 
conducted  prior  to  finalizing  the  appraisal  schedule,  and  might  lead  to  a  “go/no  go  decision 
for  the  appraisal.  The  appraisal  team  will  then  review  the  initial  objective  evidence  in  more 
detail  (typically  toward  the  end  of  the  team  training  event)  to  begin  formulating  plans  for 
collecting  missing  evidence  and  for  verifying  the  entire  data  set.  This  preliminary  readiness 
review  is  the  basis  for  the  data  collection  plan,  described  in  the  next  process,  2.1.5  Prepare 
for  Collection  of  Objective  Evidence.  The  appraisal  team  leader  generates  a  list  of  additional 
information  needed. 

The  results  of  the  analysis  of  initial  objective  evidence  are  documented  as  an  input  to  the  data 
collection  plan.  The  use  of  an  integrated  appraisal  tool  to  annotate  the  set  of  initial  objective 
evidence  will  permit  the  automated  tracking  of  information  needs  and  aid  in  the  compilation 
of  a  detailed  data  collection  plan.  When  the  completeness  of  initial  objective  evidence  is 
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insufficient  to  conduct  the  appraisal  under  the  original  schedule,  the  results  of  this  activity 
form  an  important  basis  for  renegotiating  the  appraisal  schedule. 

The  adequacy  of  objective  evidence  relative  to  model  practices  is  typically  determined  using 
a  software  tool  of  some  sort,  either  one  built  for  use  on  appraisals  or  a  spreadsheet  template. 
However,  paper  forms  and  wall  charts  can  be  used  if  preferred. 

2.1 .5  Prepare  for  Collection  of  Objective  Evidence 

Purpose— Plan  and  document  specific  data  collection  strategies,  including  the  following: 

•  sources  of  data 

•  tools  and  techniques  to  be  used 

•  contingencies  to  manage  risk  of  insufficient  data 

Entry  Criteria 

•  Sponsor  commitment  to  proceed  with  the  appraisal  has  been  documented. 

•  Appraisal  objectives  and  constraints  have  been  documented. 

•  Initial  data  have  been  received  and  analysis  has  been  completed. 

Inputs 

•  appraisal  plan 

•  PIIs  for  the  appraised  organization 

•  initial  objective  evidence  review 

•  data  collection  status 

•  People  CMM  survey  results,  if  the  survey  was  used 

Activities —  The  three  activities  listed  below  are  required  for  this  process. 

2. 1.5.1  Perform  Readiness  Review 

2. 1.5.2  Prepare  Data  Collection  Plan 

2.1.5. 3  Replan  Data  Collection 

Outputs 

•  confirmation  that  objective  evidence  collected  is  sufficient  to  proceed 

•  initial  data  collection  plan 

•  updates  to  the  plan  as  required 

Outcome — A  finalized  data  collection  plan  to  make  team  members  aware  of  data  needs  and 
the  status  of  initial  data  available  to  them. 
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Exit  Criteria — All  preparations  for  data  collection  by  the  team  have  been  made,  and  the  data 
collection  plan  has  been  documented. 

Key  Points— The  data  collected  is  the  most  important  input  the  team  receives.  Careful 
planning,  disciplined  tracking  against  the  plan,  and  effective  corrective  actions  are 
cornerstones  to  success  in  this  process. 

Tools  and  Techniques — Using  a  spreadsheet  to  record  and  track  the  data  collection  plan  is 
common.  A  matrix  showing  the  practices  of  the  model  (or  questions  to  be  asked)  on  the 
vertical  axis  and  the  sources  of  information  on  the  horizontal  axis  provides  a  simple  planning 
and  tracking  tool. 

Metrics — Examples  include  the  following: 

•  estimated  and  tracked  calendar  time  and  effort  for  this  activity 

•  planned  and  actual  number  of  data  sources  per  practice 

•  planned  and  tracked  number  of  scripted  questions  per  interview 

•  planned  and  tracked  number  of  scripted  questions  per  PA 

•  percentage  of  planned  coverage  achieved,  per  data  collection  event  or  PA 

Metrics  are  captured  in  the  appraisal  plan. 

Verification  and  Validation — The  data  collection  plan  should  be  summarized  and  reviewed 
with  the  team  to  ensure  that  appraisal  requirements  are  successfully  implemented  if  the  plan 
is  carried  forward.  Experienced  Lead  Appraisers  will  use  historical  data  to  assess  the 
feasibility  of  (and  risks  associated  with)  the  data  collection  plan. 

Records — Planned  and  actual  coverage  of  practices  and  PAs  across  the  set  of  data  collection 
activities  should  be  recorded.  These  data  support  future  estimates  and  corrective  actions 
during  the  data  collection  activities. 

Tailoring— Replanning  is  performed  only  when  the  status  of  the  appraisal  conduct  indicates 
the  need  to  do  so. 

Additional  planning  and  coordination  steps  might  be  necessary  when  data  collection 
activities  occur  at  geographically  distributed  sites.  SCAMPI  with  People  CMM  allows 
flexibility  in  the  strategies  used  to  accomplish  the  necessary  data  collection.  The  relative 
emphasis  on  different  data  sources  and  types  can  be  tuned  to  support  buy-in,  coverage,  and 
rigor  for  important  areas. 

Interfaces  with  Other  Processes — The  data  collection  plan  is  an  essential  element  of  the 
.appraisal  plan.  It  is  created  after  the  initial  objective  evidence  is  analyzed  and  contains  a  set 
of  strategies  for  collecting  the  data  needed  to  meet  the  objectives  of  the  appraisal.  The  data 
collection  plan  is  reviewed  and  revised  on  a  continual  basis  throughout  the  appraisal. 
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Processes  critical  to  the  success  of  the  appraisal  include  dynamically  managing  the  inventory 
of  data  on  hand,  the  list  of  data  needed,  and  the  available  data  collection  opportunities 

Summary  of  Activities — The  activities  in  this  process  serve  to  (a)  establish  the  initial 
planning  baseline  for  the  acquisition  of  objective  evidence  and  (b)  update  the  plan  to  account 
for  information  acquired  and  unexpected  developments.  Since  SCAMPI  with  People  CMM  is 
a  data-intensive  method,  the  conduct  of  these  activities  in  accordance  with  the  descriptions 
provided  is  essential  to  the  successful  use  of  the  appraisal  method. 

2.1. 5.1  Perform  Readiness  Review 

Activity  Description — The  available  objective  evidence  is  reviewed  to  determine  how  much 
of  the  requested  objective  evidence  has  been  gathered  and  whether  it  is  sufficient  to  proceed 
or  if  replanning  is  required. 

Required  Practices 

•  Determine  whether  the  objective  evidence  for  each  process  instance  is  adequate  to 
proceed  with  the  appraisal  as  planned. 

•  Review  the  feasibility  of  the  appraisal  plan  in  light  of  the  inventory  of  objective  evidence 
available. 

The  decision  after  review  is  captured  in  the  appraisal  plan. 

Parameters  and  Limits — At  least  one  readiness  review  must  be  conducted  and  enough  time 
allowed  for  the  team  to  make  use  of  the  information  before  the  team  is  assembled  on  site  for 
data  collection. 

Objective  evidence  for  all  PAs  within  the  scope  of  the  appraisal  and  all  units  sampled  to 
represent  the  appraised  organization  must  be  reviewed.  In  small  organizations  or  in 
organizations  with  few  units,  sampling  should  not  be  necessary.  In  large  organizations, 
sampling  might  be  a  pragmatic,  if  not  necessary,  solution  to  controlling  costs  in  a  Class  A 
appraisal.  An  organization  might  have  only  one  unit,  or,  at  the  other  extreme,  too  many  units 
to  perform  an  appraisal  for  a  reasonable  investment  of  time  and  money.  All  sampling  should 
be  random,  but  selected  units  should  represent  the  demographics  of  the  organization  (i.e.,  all 
units  within  the  scope  of  the  appraised  organization  are  included  in  the  sampling  approach). 
Sampled  units  are  represented  by  the  interview  of  the  managers  selected  (sampled).  More 
units,  if  not  all,  can  be  represented  when  the  individual  contributors  (ICs)  are  randomly 
selected  for  the  IC  interviews.  At  a  minimum  the  sampled  units  will  have  ICs  interviewed. 
However,  in  most  cases  ICs  will  represent  more  units  than  those  sampled.  Refer  to  Appendix 
A  for  more  information  about  sampling. 

While  objective  evidence  in  the  form  of  documents  is  required  during  the  initial  document 
review  (IDR),  it  might  also  be  requested  during  the  interviews  and  might  come  from  units  not 
sampled. 
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In  large  organizations,  workgroups  within  units  will  also  be  sampled.  Again,  the  sampling 
should  be  random  and  representative  of  the  unit  in  which  the  workgroups  reside. 

More  workgroups  are  represented  when  interviewing  the  ICs.  At  a  minimum  the  selected 
(sampled)  workgroups  are  represented,  but  other  workgroups  should  participate  in  the  IC 
interviews.  This  is  especially  important  in  large  organizations.  While  objective  evidence  in 
the  form  of  documents  is  not  required  for  these  non-selected  workgroups,  it  may  be  requested 
during  the  interviews. 

Optional  Practices — Integrating  a  readiness  review  with  the  team  training  event  will  allow 
the  appraisal  team  leader  to  help  the  team  understand  the  data  available  to  support  the 
appraisal. 

Implementation  Guidance — A  summary  of  the  inventory  of  objective  evidence  and 
readiness  to  proceed  should  be  reviewed  with  the  sponsor  or  his  or  her  designee.  If 
insufficient  objective  evidence  is  available,  the  appraisal  team  leader  might  need  to  initiate 
replanning  in  light  of  newly  discovered  constraints.  See  activity  2. 1.1. 2  Determine  Appraisal 
Constraints.  The  criteria  for  adequacy  will  depend  on  where  the  readiness  review  occurs  in 
the  schedule  and  the  degree  of  verification  versus  discovery  that  is  being  sought  for  the  on¬ 
site  phases  of  the  appraisal. 

More  than  one  readiness  review  will  likely  be  needed.  The  first  one  should  be  performed 
early  in  the  planning  phase  and  the  second  once  the  objective  evidence  has  been  gathered  and 
the  appraisal  is  ready  to  start.  This  review  might  be  conducted  in  conjunction  with  the  team¬ 
training  event. 

Thresholds  for  the  sufficiency  of  data  should  be  established  as  targets  for  the  readiness 
review.  For  example,  an  80%  threshold  might  be  used  to  initiate  replanning  at  the  final 
readiness  review.  That  is,  the  appraisal  team  leader  establishes  an  expectation  with  the 
sponsor  that  if  more  than  20%  of  the  objective  evidence  is  missing  at  the  time  of  team 
training,  the  appraisal  will  need  to  be  replanned.  However,  the  primary  goal  is  to  reduce  the 
risk  that  there  will  be  insufficient  objective  evidence  to  make  the  determinations  required  by 
the  appraisal  plan  in  the  time  allotted.  Thus,  whenever  possible  receiving  100%  of  objective 
evidence  prior  to  interviews  should  be  the  goal.  Thresholds  are  included  in  the  appraisal  plan. 

The  readiness  review  is  a  key  event  whose  impact  should  not  be  underestimated.  Failure  to 
adequately  review  the  objective  evidence  available  and  determine  the  impact  on  the  appraisal 
plan  can  have  grave  consequences  for  the  appraisal  team  during  the  on-site  period.  This 
might  include  long  hours,  exhaustion,  extensive  ad  hoc  data  collection  (discovery),  or  the 
inability  to  achieve  appraisal  objectives  within  defined  estimates  and  constraints. 

2.1 .5.2  Prepare  Data  Collection  Plan 

Activity  Description — The  data  collection  activities  are  tailored  to  meet  the  needs  for 
objective  evidence  so  that  the  extent  of  practice  implementation  can  be  determined. 
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For  practices  that  have  objective  evidence,  a  strategy  for  verifying  evidence  is  formulated. 

For  practices  that  lack  objective  evidence,  a  strategy  for  discovering  evidence  is  formulated. 

The  data  collection  plan  is  typically  embodied  in  a  number  of  different  artifacts  used  during 
the  appraisal  process.  The  appraisal  plan  includes  information  about  the  site,  units,  and 
participants  involved  in  the  appraisal. 

This  is  the  highest  level  of  information  that  helps  document  and  communicate  the  data 
collection  plan.  Detailed  information  on  data  collection  can  be  recorded  in  work  aids  that 
manage  appraisal  data  and  in  the  appraisal  schedule.  A  record  of  “information  needed”  items 
is  the  most  detailed  representation,  while  document  lists,  interview  schedules,  and  the 
assignment  of  PA  mini-teams  help  shape  the  strategy  for  obtaining  the  needed  data. 

Required  Practices 

•  Determine  participants  for  interviews. 

•  Determine  artifacts  to  be  reviewed. 

•  Determine  presentations  and  demonstrations  to  be  provided  (e.g.,  process  owners  can 
present  the  rationale  behind  the  approaches  taken  by  the  organization,  and 
demonstrations  can  be  given  for  online  appraisal  systems,  knowledge  management 
systems,  and  skills  databases). 

•  Determine  team  roles  and  responsibilities  for  data  collection  activities. 

•  Document  the  data  collection  plan. 

Parameters  and  Limits — For  every  instantiation  of  every  model  practice,  the  data  collection 
plan  must  specify  how,  when,  and  by  whom  the  objective  evidence  will  be  verified. 

For  instantiations  of  model  practices  that  have  not  been  addressed  in  the  initial  objective 
evidence,  the  data  collection  plan  must  specify  how  the  team  intends  to  discover  the  presence 
or  absence  of  objective  evidence  that  characterizes  the  extent  of  implementation  for  that 
practice. 

The  data  collection  plan  (often  documented  in  a  variety  of  artifacts)  includes  the  following: 

•  assignment  of  PAs  to  team  members  and  mini-teams 

•  summary  of  initial  objective  evidence  provided  by  the  organization 

•  identification  of  highest  priority  data  needs 

•  initial  allocation  of  data  needs  to  data-gathering  events 

•  identification  of  instruments  to  be  administered 

•  identification  of  participants  to  be  interviewed 

•  interview  schedule,  revised  to  include  more  detail 

•  identification  of  a  starter  set  of  interview  questions 

•  identification  of  documents  still  needed  (if  any) 
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•  risks  associated  with  the  sufficiency  of  the  data  and  the  adequacy  of  the  schedule 

Optional  Practices — Review  the  status  of  the  objective  evidence  database  with  members  of 
the  appraised  organization  to  elicit  additional  objective  evidence  or  to  expand  on  the  evidence 
available.  This  allows  the  appraisal  team  leader  to  validate  the  data  collection  plan  to  some 
extent. 

Implementation  Guidance — Sources  of  objective  evidence  include  instruments,  documents, 
presentations,  and  interviews  (see  process  2.2. 1  Examine  Objective  Evidence).  Objective 
evidence  is  differentiated  in  terms  of  PH  type  (e.g.,  direct  artifacts,  indirect  artifacts,  and 
affirmations)  as  described  in  activity  2. 1.4.5  Obtain  Initial  Objective  Evidence.  A 
combination  of  these  indicator  types  is  required  for  corroboration  (see  activity  2.2.2. 1 
Examine  Objective  Evidence).  The  data  collection  status  is  continually  monitored  during 
appraisal  activities  (see  process  2.2.3  Document  Objective  Evidence)  to  ensure  that  sufficient 
data  coverage  is  obtained.  These  are  all  key  considerations  that  should  be  understood  and 
accounted  for  in  the  generation  of  the  data  collection  plan. 

Multiple  types  of  interviews  can  be  used  to  obtain  face-to-face  affirmations  (see  activity 
2.2. 1.4  Examine  Objective  Evidence  from  Interviews): 

•  standard  structured  interviews  scheduled  in  advance  and  using  scripted  questions 

•  on-call  interviews,  scheduled  in  advance  for  calendar  purposes,  but  held  only  if  necessary 

•  office  hours  interviews,  for  which  interviewees  are  notified  that  they  might  need  to  be 
available  as  a  contingency  during  scheduled  periods 

A  robust  data  collection  plan  will  provide  for  all  three  types  of  interviews.  Start  with  a  full  set 
of  scheduled  interviews  early  in  the  planning  phase  and  gradually  add,  eliminate,  or  modify 
them  as  the  inventory  of  initial  objective  evidence  indicates  the  need.  The  mini-teams  might 
conduct  office  hours  interviews,  even  during  team  training,  to  more  fully  populate  the 
inventory  of  objective  evidence  prior  to  the  start  of  the  on-site  data  collection  activities. 

Planning  for  document  reviews  should  include  organizational-,  unit-,  and  implementation- 
level  artifacts,  as  described  in  activity  2.2. 1.3  Examine  Objective  Evidence  from  Documents. 

Ultimately,  the  appraisal  team  will  need  to  have  data  on  each  practice  in  the  People  CMM 
model  for  each  organizational  element  in  the  appraisal  scope.  For  PAs  addressing  practices 
implemented  at  the  unit  level  (e.g.,  training  and  development),  data  on  each  instantiation  of 
the  practice  will  be  collected.  For  PAs  addressing  practices  implemented  at  the  organization 
level  (e.g.,  competency  analysis),  only  one  instantiation  of  each  practice  might  be  needed, 
depending  on  the  way  the  organization  chooses  to  implement  such  practices. 

Presentations  can  be  provided  by  process  owners  (POs)  for  their  respective  PAs.  While 
presentations  can  be  useful  if  they  address  the  points  in  the  model’s  PAs,  POs  might  have 
other  agendas  and  the  presentation  might  not  sufficiently  address  the  practices.  Therefore, 
either  the  PO  should  be  provided  with  a  format  for  addressing  practices,  or  sufficient  time  at 
the  end  of  the  presentation  should  be  allocated  for  appraisal  team  members  to  ask  questions 
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to  achieve  coverage.  Presentations  should  be  affirmations  of  direct  and  indirect  evidence 
already  provided,  but  the  appraisal  team  member  must  ensure  that  coverage  is  achieved. 

The  results  of  the  analysis  of  initial  objective  evidence  are  used  to  determine  which  practices 
are  not  already  covered  with  objective  evidence.  Practices  for  which  no  initial  objective 
evidence  has  been  provided  should  be  identified  as  high-risk  areas  for  the  team  to  address 
immediately.  The  schedule  for  data  collection  might  need  to  change  dramatically  if  the  team 
is  unable  to  find  relevant  data  for  these  areas  in  short  order.  For  practices  that  have  data 
available  in  the  initial  objective  evidence,  the  team  members  assigned  to  the  PAs  plan  the 
strategy  for  verifying  the  implementation  of  each  of  the  practices  through  review  of  the 
named  documents,  interviews  with  the  people  who  play  the  named  roles,  or  other  data 
collection  events.  Artifacts  used  to  manage  data  collection  events  are  populated  with  the 
current  understanding  of  the  planned  data  collection  events,  as  listed  below. 

•  The  schedule  for  interviews  is  finalized  so  participants  can  be  informed  that  they  are 
expected  to  participate. 

•  The  list  of  documents  on  hand  (or  accessible  electronically)  is  finalized  so  the  team 
members  know  what  is  and  what  is  not  available  for  document  review. 

•  A  preliminary  allocation  of  practices  to  be  covered  in  each  of  the  scheduled  interviews  is 
documented. 

•  A  list  of  needed  documents  (not  yet  available  to  the  team)  is  generated,  if  there  are  any 
known  needs  for  documents  at  this  point. 

2.1 .5.3  Replan  Data  Collection 

Activity  Description— The  data  collection  plan  is  updated  as  required  during  the  conduct  of 
the  readiness  review  or  during  the  appraisal  itself  as  objective  evidence  is  found  or  new 
sources  of  information  are  uncovered.  The  activity  described  in  this  section  refers  to  a  more 
substantial  change  in  the  plan,  which  is  expected  to  be  rare  in  practice.  If  during  the  conduct 
of  an  appraisal,  the  team  discovers  that  their  assumptions  about  the  availability  of  objective 
evidence  are  substantially  incorrect,  the  appraisal  team  leader  might  renegotiate  the  appraisal 
plan  with  the  sponsor. 

Required  Practices 

•  Review  the  current  inventory  of  objective  evidence  and  determine  model  practices  for 
which  the  objective  evidence  is  inadequate  relative  to  the  appraisal  plan. 

•  Revise  the  data  collection  plan  as  necessary  based  on  the  appraisal  status  and  availability 
of  objective  evidence. 

•  Renegotiate  the  appraisal  plan  with  the  sponsor  if  the  appraisal  cannot  proceed  as 
planned. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 
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Parameters  and  Limits— This  activity  is  not  a  substitute  for  tactical  decisions  about  where 
and  how  to  find  objective  evidence.  The  intent  of  this  activity  is  to  respond  to  a  major  gap 
between  expected  data  and  actual  data. 

Major  gaps  between  expected  and  actual  data  might  occur,  for  example,  as  a  result  of  the 
following: 

•  inaccurate  assumptions  about  the  availability  of  objective  evidence 

•  content  of  artifacts  or  information  from  interviews  not  providing  significant  amounts  of 
the  information  required  and  other  sources  not  being  planned 

•  unexpected  absence  of  multiple  key  interviewees 

•  unanticipated  delays  in  the  implementation  of  new  processes 

•  major  customer-driven  emergencies  for  one  or  more  of  the  sampled  units 

Optional  Practices — Risk  analysis  can  be  conducted  during  early  planning  activities  to 
establish  thresholds  and  limits  for  the  amount  of  missing  objective  evidence  that  will  trigger 
this  activity.  This  enables  the  appraisal  team  leader  to  state,  in  advance,  the  conditions  under 
which  the  team  and  the  sponsor  must  renegotiate  the  appraisal  plan. 

Contingency  planning  done  in  advance  to  identify  ways  of  overcoming  issues  associated  with 
missing  objective  evidence  could  include  the  following: 

•  an  alternate  (fall-back)  schedule  for  the  appraisal 

•  staffing  to  conduct  a  “crash  data  collection”  activity 

•  reducing  the  scope  of  the  appraisal  (e.g.,  appraising  fewer  PAs,  limiting  the  extent  of  the 
appraised  organization  appraised) 

Implementation  Guidance — This  activity  serves  as  a  “pressure  valve”  of  sorts  for  the 
appraisal.  The  pressure  to  perform  the  appraisal  under  unrealistic  conditions  can  lead  to  a 
severe  degradation  in  the  quality  of  the  appraisal  outputs.  Carefully  planning  for 
contingencies  and  communicating  them  to  the  sponsor  help  to  protect  the  standards  that  must 
be  met  in  the  performance  of  an  appraisal.  Clearly  documenting  the  data  collection  plan  and 
regularly  monitoring  the  availability  of  data  compared  to  the  plan  support  effective  risk 
mitigation. 

When  this  activity  must  be  employed  to  recover  from  an  unrealistic  expectation,  the 
documentation  reflecting  the  assumptions  made  during  planning,  as  well  as  concrete  facts 
about  what  is  or  is  not  available,  are  used  to  renegotiate  with  the  appraisal  sponsor.  This  is 
one  reason  a  detailed  appraisal  plan,  with  the  sponsor’s  signature,  is  a  required  artifact  for  the 
conduct  of  a  SCAMPI  with  People  CMM  appraisal. 
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2.2  Phase  2 


This  phase  has  four  processes:  Examine  Objective  Evidence,  Verify  and  Validate  Objective 
Evidence,  Document  Objective  Evidence,  and  Generate  Appraisal  Results. 

2.2.1  Examine  Objective  Evidence 

Purpose — Collect  information  about  the  practices  implemented  in  the  organization  and  relate 
that  data  to  the  reference  model  in  accordance  with  the  data  collection  plan.  Take  corrective 
actions  and  revise  the  data  collection  plan  as  needed. 

Entry  Criteria 

•  Data  collection  has  been  planned. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

•  Participants  have  been  informed  about  the  appraisal  process  and  their  roles  in  it. 

Inputs 

•  appraisal  data 

-  initial  objective  evidence 

-  documents 

-  documented  practice  implementation  gaps,  if  any 

-  feedback  on  preliminary  findings  (if  that  point  in  the  timeline  has  been  reached) 

•  data  collection  plan 

-  appraisal  schedule 

-  interview  schedule 

-  document  list 

-  new  interview  questions 

Activities — The  four  activities  listed  below  are  required  for  this  process. 

2.2. 1.1  Examine  Objective  Evidence  from  Instruments 

2.2. 1 .2  Examine  Objective  Evidence  from  Presentations 

2.2. 1.3  Examine  Objective  Evidence  from  Documents 

2.2. 1.4  Examine  Objective  Evidence  from  Interviews 
Outputs 

•  updated  appraisal  data 

•  updated  data  collection  plan 
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Outcome — After  the  final  iteration  of  this  process,  the  team  has  sufficient  data  to  create 
appraisal  findings  and  to  make  judgments  about  the  implementation  of  practices,  as  well  as 
the  satisfaction  of  goals. 

Exit  Criteria— The  reference  model  and  the  organizational  scope  have  been  covered,  and  the 
team  is  ready  to  produce  the  appraisal  outputs. 

Key  Points — The  efficient  collection  of  objective  evidence  results  from  the  careful  creation 
and  execution  of  the  data  collection  plan.  Key  points  to  consider  are  effective  contingency 
planning  and  the  use  of  work  aids  to  monitor  progress.  The  team  must  be  able  to  focus  on 
examining  the  most  relevant  information  available  rather  than  be  distracted  by  rooting  out 
new  evidence. 

Tools  and  Techniques— Wall  charts  and  other  visual  aids  are  often  used  to  display  the  results 
of  data  collection  activities.  Electronic  tools  are  prevalent  among  experienced  Lead 
Appraisers  and  can  be  very  effective  for  continually  monitoring  and  updating  the  inventory  of 
objective  evidence. 

Metrics — Tracking  the  actual  coverage  obtained  against  the  planned  coverage  in  each  data 
collection  activity  facilitates  timely  corrective  actions  where  they  are  needed.  The  most 
critical  resource  during  an  appraisal  is  time.  A  timekeeper  provides  feedback  on  team 
performance  during  data  collection  and  verification  activities.  Recording  the  actual  duration 
of  planned  events  helps  the  team  recover  from  unexpected  events. 

Metrics  are  captured  during  the  appraisal. 

Verification  and  Validation— The  appraisal  method  provides  detailed  verification  and 
validation  procedures  for  objective  evidence.  The  procedures  are  described  in  process  2.2.2 
Verify  and  Validate  Objective  Evidence. 

Records— Work  aids  used  to  record  and  track  the  progress  of  data  collection  activities  are 
retained  for  traceability  and  provide  an  important  input  to  a  final  report  describing  the 
appraisal,  if  the  sponsor  requests  a  final  report.  The  duration  and  effort  required  for  specific 
data  collection  events  can  be  recorded  to  provide  historical  data  useful  for  planning 
subsequent  appraisals. 

Tailoring — The  method  is  flexible  in  the  use  of  customized  data  collection  instruments, 
presentations,  document  reviews,  and  interviews.  Specialized  forms  of  these  data  collection 
methods  can  be  constructed  to  meet  the  objectives  of  the  appraisal.  Rather  than  a 
standardized  questionnaire,  an  organization-specific  questionnaire  could  be  used  that  contains 
local  jargon.  However,  this  must  be  done  with  care  to  ensure  the  questionnaires  do  not  lead 
.respondents  to  the  preferred  answers.  Standardized  presentations  can  be  employed  to  provide 
the  team  with  an  “inbrief’  at  the  start  of  the  appraisal  (e.g.,  PO  presentations  or 
demonstrations  of  tools  or  databases).  The  method  also  provides  flexibility  in  choosing  the 
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number,  duration,  style,  and  make-up  of  interview  sessions  within  specified  boundaries  and 
allows  telephone  and  video  conferences. 

Interfaces  with  Other  Processes— The  activities  that  provide  the  team  with  data  needed  to 
produce  reliable  appraisal  outputs  are  perhaps  the  most  visible  part  of  the  appraisal  process 
for  the  appraised  organization.  For  this  reason,  SCAMPI  with  People  CMM  places  a  heavy 
emphasis  on  methodically  planning  and  tracking  the  data  collected  during  an  appraisal.  The 
initial  objective  evidence  collected  early  in  the  process  allows  team  members  to  analyze  the 
state  of  information  available  at  the  earliest  stages  of  the  appraisal  and  narrow  their  search  for 
new  information.  An  explicit  understanding  of  what  information  is  needed  and  how  that 
information  will  be  used  drives  the  activities  associated  with  this  process. 

Summary  of  Activities — Team  members  continually  manage  the  data  collected  previously 
and  work  to  fill  known  information  needs.  Instruments  tend  to  be  used  early  in  the  appraisal 
process  to  provide  leads  for  other  data  collection  activities  and  affirmations  of  implemented 
practices. 

Presentations  are  sometimes  used  to  provide  a  flexible  forum  where  members  of  the 
organization  can  provide  important  information  about  the  practices  implemented  in  the 
organization. 

Documents  are  the  most  explicit  and  lasting  representation  of  practice  implementation  in  the 
organization,  and  the  team  uses  them  to  understand  how  practices  in  the  People  CMM  model 
are  implemented.  Interviews  are  used  as  the  most  dynamic  data  collection  technique, 
allowing  for  branching  among  related  topics  and  the  explanation  of  contextual  information 
that  affects  the  implementation  of  practices  and  alternative  practices.  The  appraisal  activities 
conducted  for  each  of  these  data  collection  sources  are  listed  below. 

•  Determine  if  the  information  obtained  is  acceptable  as  objective  evidence. 

•  Relate  the  objective  evidence  to  corresponding  practices  in  the  appraisal  reference  model. 

•  Relate  the  objective  evidence  to  the  appropriate  part  of  the  appraised  organization  (i.e., 
the  instantiation). 

2.2.1. 1  Examine  Objective  Evidence  from  Instruments 

Activity  Description — Instruments  provided  by  the  appraised  organization  are  reviewed  to 
obtain  objective  evidence  reflecting  the  organization’s  implementation  of  model  practices 
(e.g.,  the  administered  survey  results).  Instruments  include  questionnaires,  surveys,  and  other 
written  information  that  indicates  practice  implementation. 

This  activity  builds  on  the  inventory  of  objective  evidence  developed  during  appraisal 
planning  and  preparation.  The  appraisal  team  considers  the  information  contained  in  the 
instruments  and  determines  if  it  is  accurate,  consistent,  and  relevant  to  the  scope  of  the 
reference  model  being  examined. 
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Objective  evidence  obtained  from  instruments  and  from  other  sources  is  verified  in  2.2.2 
Verify  and  Validate  Objective  Evidence  and  documented  in  process  2.2.3  Document 
Objective  Evidence. 

Required  Practices 

•  Review  information  obtained  from  instruments  and  determine  if  it  is  acceptable  as 
objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence  obtained  from 
instruments. 

•  Determine  the  portions  of  the  appraised  organization  that  correspond  to  the  objective 
evidence  obtained  from  instruments. 

•  Develop  scripts  after  review  of  the  survey  results. 

All  required  practices  performance  decisions  regarding  use  of  instruments  are  captured  in  the 
appraisal  plan. 

Parameters  and  Limits — At  least  one  instrument  must  be  used  during  the  conduct  of  the 
appraisal.  (Refer  to  2.1.4.3  Administer  Instruments  for  a  description  of  instruments.)  For 
SCAMPI  with  People  CMM,  the  standard  SEI  set  of  questions  for  the  survey  can  be  used. 
Also  the  evidence  database,  instantiated  from  the  PIIs  for  People  CMM,  can  be  used  as  an 
instrument. 

Optional  Practices — Create  and  administer  a  specialized  questionnaire  tailored  to  the 
Characteristics  of  the  organization  or  the  objectives  of  the  appraisal.  The  People  CMM  survey 
could  also  be  used. 

Implementation  Guidance — Using  instruments  to  gather  written  information  from  members 
of  the  organization  is  a  relatively  low-cost  data  collection  technique,  when  done  well.  Data  of 
this  type  tend  to  be  most  useful  when  provided  early  in  the  appraisal  conduct  and  can  lead  to 
valuable  insights  about  the  location  of  the  data  during  subsequent  data  collection  events. 

The  most  common  instrument  used  is  the  organization’s  PH  database,  which  provides 
traceability  of  reference  model  practices  to  the  processes  and  work  products  implemented 
within  the  organization.  If  organizations  have  not  yet  implemented  this  asset,  a  questionnaire 
can  be  used  to  gather  closed-ended  responses  and  comments  about  the  implementation  of 
each  model  practice  in  each  sampled  unit  in  the  appraised  organization. 

It  is  also  the  responsibility  of  the  appraisal  team  leader  to  prevent  duplicate  data  entry  on 
multiple  instruments.  No  oiganization  should  be  asked  to  provide  the  same  information  in 
two  or  more  formats. 

Activities  performed  during  the  survey  provide  appraisal  team  members  with  information 
about  the  consistency  of  workforce  practices  and  the  major  issues  related  to  them.  The  survey 
responses  provide  guides  for  workforce  practices  or  issues  that  should  be  probed  during 


60 


CMU/SEI-2005-SR-001 


document  review  and  interviews.  They  also  provide  data  to  corroborate  information  found  in 
other  data-gathering  tasks.  Activities  that  constitute  this  task  were  discussed  in  task  2. 1.4.4 
Analyze  People  CMM  Survey  Results.  If  these  activities  have  already  been  completed  prior 
to  the  on-site  appraisal,  this  task  might  be  shortened  and  only  involve  a  short  review  of  any 
issues  related  to  survey  results. 

Activities  performed  during  this  task  result  in  the  creation  of  scripts  to  guide  the  interviews 
of  process  owners  and  managers  and  the  discussions  with  the  workforce.  Once  the  appraisal 
team  consolidates  data  from  the  survey  results  and  documents,  they  should  script  questions 
for  interviews  to  gather  further  data  that  corroborate  strengths  and  weaknesses.  These 
scripted  questions  allow  the  appraisal  team  to  probe  for  more  information  about  their 
preliminary  observations  and  elicit  further  information  when  needed  during  the  on-site 
interviews.  Generally,  in  each  management  interview  or  workforce  discussion,  at  least  one 
question  should  be  asked  about  each  goal  in  each  process  area  in  the  scope  of  the  appraisal. 
Additional  questions  should  be  asked  in  those  areas  where  there  is  greater  uncertainty 
regarding  the  consistency  of  the  practices.  The  appraisal  team  has  the  option  of  developing 
interview  scripts  for  the  process  owners,  managers,  and  workforce  discussions  during  the 
same  session  or  in  separate  sessions  held  prior  to  each  type  of  interview.  If  the  appraisal  team 
develops  all  scripts  in  a  single  scripting  session,  they  should  review  the  scripts  immediately 
before  each  type  of  interview  to  find  out  if  the  information  being  collected  needs  to  be 
adjusted. 

Some  of  the  activities  typically  performed  during  this  task  are  listed  below. 

•  Write  question  scripts  for  each  process  owner  to  be  interviewed. 

•  Write  a  question  script  to  be  used  during  manager  interviews. 

•  Write  a  question  script  to  be  used  during  workforce  discussions. 

•  Review  interview  scripts  for  coverage,  time,  and  information  needed. 

•  Determine  interview  roles. 

•  Revise  scripts  based  on  results  of  previous  interviews. 

Note  that  scripting  goes  on  throughout  the  appraisal.  Scripts  are  modified  based  on  the 
coverage  results  and  information  needed  to  complete  coverage. 

2.2.1 .2  Examine  Objective  Evidence  from  Presentations 

Activity  Description — Demonstrations  of  online  tools  or  libraries  to  be  accessed  by  the 
appraisal  team  are  often  the  best  way  for  members  of  the  team  to  find  data  and  information. 
The  history  of  process  improvement  in  the  organization  or  the  status  of  current  improvement 
units  can  sometimes  be  best  conveyed  to  the  appraisal  team  in  the  form  of  a  presentation. 
While  the  amount  of  data  to  be  collected  using  presentations  is  minimal,  the  ability  to  receive 
information  and  ask  questions  in  real  time  makes  this  a  valuable  data  collection  technique. 
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Objective  evidence  obtained  from  presentations  and  from  other  sources  is  documented  in 
process  2.2.3  Document  Objective  Evidence  and  verified  in  process  2.2.2  Verify  and  Validate 
Objective  Evidence. 

Required  Practices 

•  Receive  presentations  from  the  appraised  organization,  if  applicable. 

•  Review  information  obtained  from  presentations,  and  determine  if  it  is  acceptable  as 
objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence  obtained  from 
presentations. 

•  Determine  the  portions  of  the  appraised  organization  that  correspond  to  the  objective 
evidence  obtained  from  presentations. 

All  required  practices  performance  decisions  about  the  use  of  presentations  are  captured  in 
the  appraisal  plan. 

Parameters  and  Limits — Presentations  are  not  required  in  the  data  collection  plan,  but  the 
team  may  permit  presentations  of  information  by  knowledgeable  members  of  the 
organization.  Presentations  might  or  might  not  be  required  by  the  team,  depending  on  the 
appraisal  usage  mode  and  the  appraisal  objectives. 

All  team  members  need  not  be  present  at  every  presentation,  though  it  might  be 
advantageous.  A  minimum  of  two  team  members  must  be  present  in  order  to  consider  any 
presentation  a  valid  data  collection  session.  Team  members  take  notes  during  presentations  to 
document  information  for  later  use,  as  described  in  activity  2.2.3. 1  Take/Review/Tag  Notes. 

Optional  Practices — Allow  the  organization  to  provide  presentations  or  demonstrations  of 
tools  as  a  means  of  providing  objective  evidence  about  the  implementation  of  model 
practices.  Establish  a  standardized  boilerplate  for  the  appraised  organization  or  units  in  the 
appraised  organization  to  use  in  orienting  the  appraisal  team  during  the  presentation. 

in  an 

organization  can  be  revealing  and  can  help  to  shape  further  data  collection. 

Demonstrations  of  tools  supporting  the  process  infrastructure  are  sometimes  the  most 
convenient  means  of  communicating  objective  evidence.  Tools  that  are  commonly 
demonstrated  include  the  following: 

•  metrics  database 

•  process  asset  library  and  tools 
'  •  process-related  Web  pages 

•  computer-based  training  courses  or  training  repositories 

•  risk  management  databases 


62 


CMU/SEI-2005-SR-001 


•  human  resources  databases  or  human  resources  information  systems 

•  knowledge  management  systems 

•  skills  management  databases 

An  organization’s  metrics  database  can  often  embody  the  analytical  techniques  in  use  and  the 
communication  channels  that  are  supported  across  the  appraised  organization. 

2.2.1 .3  Examine  Objective  Evidence  from  Documents 

Activity  Description — A  substantial  portion  of  the  data  used  by  appraisal  team  members  is 
derived  from  documents  they  review.  Most  of  the  direct  artifacts  used  as  indicators  of 
practice  implementation  are  documents.  Document  review  is  an  effective  way  to  gain  detailed 
insight  into  the  practices  in  use  in  the  organization.  However,  without  a  clear  focus  on  the 
data  being  sought,  document  review  can  consume  a  great  deal  of  time  as  team  members 
sometimes  attempt  to  read  everything  in  hopes  that  something  useful  will  be  discovered. 

Objective  evidence  obtained  from  documents  and  from  other  sources  is  documented  in 
process  2.2.3  Document  Objective  Evidence  and  verified  in  process  2.2.2  Verify  and  Validate 
Objective  Evidence. 

Required  Practices 

•  Establish  and  maintain  a  catalog  of  documents  used  as  a  source  of  objective  evidence. 

•  Review  information  obtained  from  documents,  and  determine  if  it  is  acceptable  as 
objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence  obtained  from 

documents.  . 

•  Determine  the  portions  of  the  appraised  organization  that  correspond  to  the  objective 
evidence  obtained  from  documents. 

All  required  practices  performance  decisions  regarding  documents  are  captured  in  the 
appraisal  plan. 

Parameters  and  Limits— All  SCAMPI  with  People  CMM  appraisals  must  use  documents  as 
a  source  of  information  about  the  extent  to  which  practices  have  been  implemented  in  the 
appraised  organization  and  the  sampled  units. 

The  catalog  should  be  sufficient  to  summarize  the  documentation  of  objective  evidence  used 
as  a  basis  for  the  appraisal  ratings  generated,  as  required  by  the  appraisal  record  described  in 
activity  2.3.2.2  Generate  Appraisal  Record. 

Much  of  the  catalog  contents  can  be  obtained  from  the  mapping  data  or  instruments  obtained 
from  the  appraised  organization,  such  as  the  PH  database,  survey,  or  questionnaires.  The 
catalog  can  be  used  to  maintain  a  list  of  documents  reviewed  or  additional  documentation 
requested  from  the  appraised  organization. 
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Optional  Practices — For  organizations  with  substantial  intranets  containing  Web-based 
document  libraries,  a  member  of  the  organization  familiar  with  the  document  library  should 
provide  a  demonstration  of  the  Web-based  tools.  Links  to  other  documents  and  other  features 
of  the  Web-based  document  library  must  be  tested  prior  to  the  team’s  use  during  the 
appraisal. 

Implementation  Guidance — One  or  more  team  members  will  seek  data  for  every  practice  in 
the  model  scope  of  the  appraisal  through  document  review.  This  does  not  require  a  unique 
document  for  every  practice,  as  any  given  document  is  likely  to  provide  data  relevant  to 
multiple  practices.  To  the  greatest  extent  possible,  the  location  of  documented  evidence 
relating  to  every  practice  should  be  recorded  in  advance  of  the  team’s  arrival  at  the  site  where 
the  appraisal  will  occur.  Organizations  with  established  improvement  infrastructures  typically 
maintain  this  type  of  information  in  order  to  track  their  improvement  efforts  against  the 
model. 

If  this  information  is  incomplete,  the  team  will  need  to  discover  the  linkages  between  the 
People  CMM  model  and  the  organization’s  implemented  practices  and  will  therefore  require 
more  time  to  perform  the  appraisal. 

Implementation  Guidance:  Three  Levels  of  Documents— Documents  reviewed  during  an 
appraisal  can  be  classified  into  different  levels:  organization,  unit,  workgroup,  and 
implementation.  By  providing  further  insight  into  the  policies  and  procedures  that  guide  the 
organization’s  processes,  organization-level  documents  sometimes  help  the  team  to  eliminate 
the  need  for  a  question  during  an  interview  or  to  sharpen  the  focus  for  a  question.  A  review  of 
these  documents  provides  a  context  for  understanding  the  expectations  placed  on  units  within 
the  organization. 

Through  the  review  of  unit-  and  workgroup-level  documents,  team  members  gain  further 
insight  into  each  scheduled  interviewee’s  role  in  the  unit  and  workgroup  they  support  and  the 
terminology  generally  accepted  within  the  organization  or  unit  and  workgroup.  This  might 
lead  to  the  refinement  or  modification  of  interview  questions. 

The  team  typically  reviews  implementation-level  documents  to  validate  information  gathered 
from  other  sources,  such  as  interviews  or  higher-level  documents.  Documents  on  this  level 
provide  an  audit  trail  of  the  processes  used  and  the  practices  performed  by  the  organization. 
The  review  of  these  documents  frequently  provides  verification  of  practices  found  in 
organization-  and  unit-level  documents. 

Activities  performed  during  this  task  provide  one  source  of  evidence  regarding  the 
organization’s  workforce  practices.  The  review  of  documents  and  artifacts  helps  the  appraisal 
team  to  do  the  following: 

•  establish  an  organizational  context  for  evaluating  workforce  practices 

•  understand  how  workforce  practices  are  supported  by  the  organization 

•  identify  workforce  practices  or  issues  to  be  probed  during  interviews 
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•  ensure  that  a  portion  of  the  observations  of  each  process  area  goal  is  supported  by  a 
review  of  the  relevant  documentation  described  in  the  People  CMM 

•  map  the  organization’s  documents  to  the  People  CMM  so  the  information  can  be  used  as 
appraisal  data 

All  collected  documents  should  be  made  available  to  the  appraisal  team  in  their  secured  work 
area  during  the  entire  on-site  appraisal  period.  If  some  important  documents  are  too  sensitive 
to  be  released  or  copied  for  use  by  the  appraisal  team,  they  can  be  reviewed  where  they  are 
maintained  or  under  the  supervision  of  the  document  owner  during  an  interview  session. 

2.2.1 .4  Examine  Objective  Evidence  from  Interviews 

Activity  Description — Interviews  are  used  to  obtain  face-to-face  affirmations  relating  to  the 
implementation  of  processes  at  the  organizational,  unit,  and  workgroup  levels. 

Interviews  are  held  with  process  owners,  managers,  and  individuals  responsible  for  the  work 
being  performed.  The  appraisal  team  uses  interviews  to  understand  how  the  processes  and 
workforce  practices  are  implemented  and  to  probe  areas  where  additional  coverage  of  model 
practices  is  needed. 

Interviews  are  a  required  component  of  a  SCAMPI  with  People  CMM  appraisal.  The  criteria 
for  the  amount  of  face-to-face  evidence  to  be  collected  are  described  in  activity  2.2.2. 1  Verify 
Objective  Evidence.  This  drives  the  development  of  the  initial  interviewing  strategy  in  the 
data  collection  plan  described  in  activity  2. 1.5.2  Prepare  Data  Collection  Plan  and  the 
interview  scripts  prepared  in  2.2. 1.1  Examine  Objective  Evidence  from  Instruments.  A 
variety  of  interviewing  techniques  are  available,  and  the  appraisal  team  leader  works  with  the 
team  to  schedule  the  most  appropriate  interview  types. 

As  objective  evidence  is  gathered  throughout  the  appraisal,  the  data  collection  plan  is  revised. 
By  using  focused  investigation  techniques,  the  need  for  interviews  might  be  increased  or 
diminished,  as  long  as  the  criteria  for  face-to-face  affirmations  are  satisfied. 

Objective  evidence  obtained  from  interviews  and  from  other  sources  is  documented  in 
process  2.2.3  Document  Objective  Evidence  and  verified  in  process  2.2.2  Verify  and  Validate 
Objective  Evidence. 

Required  Practices 

•  Refine  the  data  collection  plan  to  determine  the  objective  evidence  that  must  be  obtained 
from  interview  participants. 

•  Revise  interview  scripts,  as  needed,  to  ensure  the  collection  of  the  objective  evidence  that 
must  be  obtained  from  interview  participants. 

•  Review  information  obtained  from  interviews  and  determine  if  it  is  acceptable  as 
objective  evidence. 
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•  Determine  the  model  practices  corresponding  to  the  objective  evidence  obtained  from 
interviews. 

•  Determine  the  portions  of  the  appraised  organization  that  correspond  to  the  objective 
evidence  obtained  from  interviews. 

All  required  practices  performance  decisions  regarding  interviews  are  captured  in  the 
appraisal  plan. 

Parameters  and  Limits — All  SCAMPI  with  People  CMM  appraisals  must  use  interviews  as 
a  source  of  information  on  the  extent  to  which  practices  have  been  implemented  in  the 
appraised  organization  and  within  the  sampled  units. 

All  interviews  must  include  at  least  two  members  of  the  appraisal  team,  designated  by  the 
appraisal  team  leader. 

Full  coverage  of  the  People  CMM  model  within  the  defined  scope,  the  appraised 
organization,  and  the  organization’s  life  cycle  must  be  achieved  with  the  objective  evidence 
considered  by  the  team.  Therefore  the  pool  of  potential  interviewees  must  cover  all  elements 
of  the  process  in  use  in  the  appraised  organization. 

Unit  and/or  program  management  personnel  are  typically  interviewed  individually,  but  in 
large  organizations  can  be  grouped  according  to  units.  The  focus  of  the  discussion  in  these 
interviews  can  be  scoped  to  a  particular  unit  rather  than  across  sampled  units.  Individuals  are 
typically  interviewed  in  a  group  discussion,  sampled  across  the  units  and  grouped  by  levels 
of  seniority  or  roles  within  the  appraised  organization.  The  focus  of  the  discussion  in  these 
interviews  will  therefore  be  scoped  to  a  particular  set  of  practices  used  across  the  units.  The 
rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be  communicated  to  every 
interviewee. 

In  a  subset  of  interview  sessions,  an  interviewee  might  have  a  reporting  relationship  with  an 
appraisal  team  member  in  attendance.  This  might  be  unavoidable  when  doing  sampling  of 
interviewees.  In  these  cases  the  appraisal  team  member  can  be  excused  by  the  ATL  for  these 
sessions  and  the  process  area  responsibilities  delegated  to  this  appraisal  team  member’s  mini¬ 
team  partner  or  another  appraisal  team  member. 

Optional  Practices — Request  that  interviewees  bring  a  document  or  other  artifact  with  them 
to  their  interviews  for  a  “show-and-tell”  style  interview.  Use  video/teleconference  technology 
to  conduct  interviews  at  a  distance.  Appraisers  should  not  rely  too  heavily  on  this  method.  If 
substantial  portions  of  the  interview  data  are  gathered  using  this  technology,  it  might  limit  the 
amount  of  information  collected. 

Implementation  Guidance — Interviews  provide  the  most  flexible  source  of  detailed  data. 
Face-to-face  interaction  with  people  who  enact  the  practices  being  investigated  allows  the 
team  to  seek  detailed  information  and  understand  the  interrelationships  among  various 
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practices.  Detailed  information  to  address  specific  data  collection  needs  can  be  sought  and 
verified  in  real  time. 

Avoid  sampling  interviewees  for  a  session  that  results  in  two  people  in  the  same  reporting 
chain  (e.g.,  a  superior  and  one  of  his  or  her  direct  reports)  in  the  same  interview  session.  This 
applies  to  members  of  the  appraisal  team  as  well.  People  who  have  this  type  of  relationship 
with  one  another  might  be  uncomfortable  being  candid  during  the  interview.  Should  an 
interviewee  and  an  appraisal  team  member  have  a  reporting  relationship,  the  appraisal  team 
member  should  be  asked  not  to  participate  in  that  session.  The  ATL  should  reassign  the 
interview  question  responsibilities  for  these  PAs  to  the  mini-team  partner,  another  appraisal 
team  member,  or  to  the  ATL. 

To  get  good  data  from  the  sessions,  individual  interviews  should  be  stratified  by  longevity  of 
service  (1-5, 5-10, 10+  years),  or  by  types  or  categories  of  employees. 

Samples  of  interviewees  are  typically  grouped  into  categories  that  roughly  correspond  to  life- 
cycle  phases,  engineering  disciplines,  organizational  groupings,  and/or  PA  affinities.  As 
stated  previously,  interviews  of  unit/program  management  personnel  are  typically  grouped  by 
unit,  while  individuals  sampled  for  a  given  interview  come  from  across  the  appraised 
organization. 

There  are  three  basic  forms  of  interviews  used  in  SCAMPI  with  People  CMM,  described 
below. 

Implementation  Guidance:  Standard  Interviews— The  most  structured  approach  is  the 
standard  interview,  which  is  scheduled  in  advance  and  uses  a  series  of  scripted  questions. 
Each  standard  interview  typically  involves  interviewees  with  similar  responsibilities  in  the 
organization  (e.g.,  process  owners,  workforce  members,  or  managers).  The  schedule  and 
location  of  each  interview  session  is  communicated  to  the  interviewees  well  enough  in 
advance  to  permit  their  attendance.  Questions  intended  to  elicit  data  about  particular  practices 
are  prepared  and  reviewed  in  advance,  and  the  team  follows  a  defined  process  for  conducting 
the  session.  The  entire  team  is  present  for  these  interviews.  Responsibility  for  tracking  the 
coverage  of  individual  PAs  is  typically  assigned  to  team  members.  A  single  questioner  might 
lead  the  interview  with  the  rest  of  the  team  listening  and  taking  notes,  or  the  responsibility  for 
asking  questions  might  be  distributed  among  the  team  members.  In  any  case,  all  team 
members  who  are  not  asking  questions  should  listen  and  take  notes  for  all  questions. 

An  alternative  is  to  have  the  team  work  in  parallel  interview  mini-sessions  (e.g.,  the  team 
might  be  partitioned  into  subsets  to  address  the  PAs  assigned  in  parallel  sessions).  All  units  in 
the  organization  are  potentials  for  interviews  via  the  manager  and  workforce  interviews. 

A  set  of  planned  interviews  is  defined  during  appraisal  planning.  As  the  appraisal  progresses 
and  the  objective  evidence  accumulates,  the  team  might  find  it  convenient  to  cancel  one  or 
more  of  these  interviews  to  use  the  time  for  other  activities.  Such  changes  in  the  data 
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collection  plan  should  be  made  in  a  way  that  does  not  violate  the  coverage  criteria  described 
in  process  2.2.2  Verify  and  Validate  Objective  Evidence. 

Implementation  Guidance:  On-Call  Interviews — A  more  flexible  approach  to  scheduling 
interviews  is  available  in  the  form  of  on-call  interviews,  a  variant  of  the  standard  interview. 
Prospective  interviewees  are  identified  and  notified  in  advance,  just  as  described  above. 
However,  the  interviews  are  only  held  if  team  members  decide  that  they  are  needed  and  that 
the  time  will  be  well  spent. 

The  prospective  interviewees  are  asked  to  block  off  a  period  of  time  and  are  informed 
whether  or  not  the  interview  session  will  actually  happen  well  ahead  of  time.  These 
interviews  need  not  include  the  entire  appraisal  team,  permitting  parallel  sessions  with 
different  interviewees.  However,  at  least  two  members  of  the  appraisal  team  (selected  by  the 
appraisal  team  leader)  must  participate. 

Implementation  Guidance:  Office  Hours  Interviews— Office  hours  interviews  represent 
an  agreement  for  availability  that  permits  pairs  of  team  members  to  visit  interviewees  at  then- 
desks,  cubicles,  or  offices.  As  with  the  on-call  interviews,  the  prospective  interviewees  block 
off  a  specific  time  period  to  be  available  on  a  contingency  basis.  Most  prospective 
interviewees  will  be  able  to  continue  with  their  daily  work  and  accommodate  an  interruption 
if  the  team  needs  to  speak  with  them.  Again,  the  interview  will  occur  only  if  specific  data 
needs  are  identified.  The  interviewees  should  be  informed  that  they  might  receive  only 
limited  advanced  notice  for  these  interviews,  although  confirming  the  interview  at  least  a  day 
in  advance  is  a  courtesy  that  should  be  offered  whenever  possible. 

Interviews  are  conducted  with  three  types  of  people  when  using  SCAMPI  with  People  CMM: 
process  owners,  managers,  and  workforce. 

Interview  Process  Owners — During  this  task,  information  is  collected  from  those  who  are 
responsible  for  workforce  practices  at  the  organizational  level.  This  information  describes  the 
workforce  practices  and  support  in  place  at  the  organizational  level.  These  interviews  provide 
the  team  with  organizational  perspective  and  do  not  necessarily  provide  information  about 
the  actual  implementation  or  consistency  of  these  practices  at  the  working  level.  These 
interviews  might  also  provide  an  opportunity  to  review  documentation  retained  within  a 
process  owner’s  area.  It  is  typically  better  to  interview  process  owners  from  different 
functions  separately  to  make  better  use  of  their  time  and  carefully  focus  the  interview 
objectives,  but  a  mixture  of  POs  can  also  be  scheduled. 

The  types  of  process  owners  typically  interviewed  include  the  following: 

•  human  resources  staff 

-•  training  department  staff 

•  compensation  specialists 

•  staffing  specialists 
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•  performance  management  specialists 

•  career  development  or  succession  specialists 

•  competency  analysts 

•  team-building  specialists 

•  building  and  maintenance  staff 

•  computing  facilities  staff 

•  interview  managers 

•  other  individuals  or  groups  as  appropriate 

Activities  performed  during  this  task  involve  collecting  information  from  those  who  are 
responsible  for  performing  workforce  practices.  Interviews  with  managers  allow  the  appraisal 
team  to 

•  determine  the  consistency  of  the  workforce  practices  used  by  managers  at  the  unit  or 
workgroup  level 

•  identify  differences  or  inconsistencies  in  workforce  practices  across  major  appraised 
organizations 

•  identify  any  unique  or  unit-  or  site-specific  workforce  management  practices 

•  understand  the  relationship  between  the  workforce  practices  supported  at  the  organization 
level  and  how  they  are  performed  at  the  unit  and  workgroup  level 

•  understand  the  middle  managers’  perspectives  on  how  workforce  practices  are  performed 

•  identify  workforce  practices  that  managers  believe  should  be  improved 

Managers  selected  for  interviews  could  be  functional  or  matrix  managers,  team  leaders, 
project  leaders,  middle  managers,  or  unit  or  workgroup  managers.  Managers  should  be 
sampled  to  obtain  adequate  representation  of  areas  and  managerial  levels  in  the  organization. 

To  allow  the  fullest  sampling  of  managers  across  the  organization,  these  interviews  should  be 
conducted  by  mini-teams  composed  of  two  to  three  appraisal  team  members.  Each  mini-team 
will  typically  conduct  two  manager  interviews.  Thus,  two  to  four  mini-teams  each  conducting 
two  interviews  will  collect  data  from  a  total  of  four  to  eight  managers.  Each  mini-team 
should  interview  one  manager  at  a  time  unless  there  is  a  sensible  reason  for  other  managers  to 
be  included,  such  as  a  shared  performance  of  workforce  practices. 

During  the  interview,  one  mini-team  member  acts  as  the  lead  interviewer  while  other  team 
members  take  notes  on  the  manager’s  responses.  The  lead  interviewer  opens  the  session  and 
explains  the  interview  context  and  confidentiality  guarantees.  The  lead  interviewer  then  asks 
questions  from  the  scripts  prepared  in  advance  and  manages  progress  to  ensure  that  all 
required  questions  are  completed  before  the  scheduled  close  of  the  interview.  Other  mini¬ 
team  members  might  ask  follow-up  questions  or  request  clarifications. 
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Following  their  initial  manager  interview,  each  mini-team  performs  a  minimal  consolidation 
of  the  data  that  they  have  collected  to  identify  areas  they  wish  to  highlight  or  de-emphasize 
during  their  second  manager  interview.  Interview  scripts  are  revised  if  necessary. 

Conduct  Workforce  Discussions — Activities  performed  during  this  task  involve  collecting 
information  from  those  who  experience  workforce  practices.  A  workforce  discussion 
typically  involves  a  group  of  8  to  12  individuals  (but  groups  can  be  larger)  who  are  invited  to 
discuss  their  experiences  in  the  organization.  These  groups  are  prompted  with  questions  from 
the  scripts  prepared  for  use  with  the  workforce,  but  the  discussion  is  free  flowing.  Workforce 
discussions  allow  the  appraisal  team  to 

•  understand  how  the  workforce  experiences  the  organization’s  workforce  management 
practices 

•  identify  workforce  practices  that  the  workforce  believes  should  be  improved 

•  determine  the  consistency  of  the  workforce  practices  used  by  managers  at  the  unit  level 

•  corroborate  data  provided  in  process  owner  and  manager  interviews 

•  identify  differences  or  inconsistencies  in  workforce  practices  across  major  appraised 
organizations 

•  identify  any  unique  unit,  workgroup,  or  site-specific  workforce  management  practices 

•  understand  the  relationship  between  the  workforce  practices  supported  at  the  organization 
level  and  how  they  are  performed  at  the  unit  or  workgroup  level 

Individuals  selected  for  workforce  discussions  can  come  from  any  job  type  included  in  the 
scope  of  the  appraisal.  Individuals  should  be  sampled  to  obtain  adequate  representation  of 
units,  assignments,  and  job  types.  Discussion  groups  could  be  organized  by  job  type,  work 
assigned,  or  some  other  relevant  characteristic  (e.g.,  junior  vs.  senior  staff). 

To  allow  the  fullest  sampling  of  the  workforce,  these  discussions  should  be  conducted  by 
mini-teams  composed  of  two  to  four  appraisal  team  members.  Each  mini-team  typically 
conducts  two  workforce  discussions.  Thus,  two  to  four  mini-teams  each  conducting  two 
workforce  discussions  will  collect  data  from  a  total  of  32  to  160  individuals. 

During  the  discussion,  one  mini-team  member  acts  as  the  lead  interviewer  while  other  team 
members  take  notes  on  the  responses.  The  lead  interviewer  opens  the  session  and  explains  the 
discussion  context  and  confidentiality  guarantees.  Participants  are  asked  not  to  discuss 
anything  they  hear  during  the  discussion  to  protect  confidentiality.  The  lead  interviewer  then 
asks  questions  from  the  scripts  prepared  in  advance  and  manages  progress  to  ensure  that  all 
required  questions  are  completed  before  the  scheduled  close  of  the  discussion.  Other  mini¬ 
team  members  might  ask  follow-up  questions  or  request  clarifications. 

'  Following  the  interview,  each  mini-team  should  perform  a  minimal  consolidation  of  the  data 
to  identify  areas  they  wish  to  highlight  or  de-emphasize  during  their  second  workgroup 
discussion.  Discussion  scripts  are  revised  if  necessary. 
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Some  of  the  activities  typically  performed  during  these  three  sets  of  interviews  are  listed 
below. 

•  Explain  confidentiality  and  purpose  of  the  discussion. 

•  Ask  scripted  questions. 

•  Record  responses. 

•  Ask  follow-up  questions  and  request  clarifications. 

•  Ask  for  final  comments. 

•  Indicate  the  time  and  place  of  participants’  next  involvement. 

Additional  activities  typically  performed  during  this  task  are  listed  below. 

•  Set  up  the  interview  room. 

•  Introduce  the  participant(s)  to  the  interview  process. 

•  Ask  scripted  questions. 

•  Record  notes. 

•  Conclude  the  interview. 

2.2.2  Verify  and  Validate  Objective  Evidence 

Purpose — Verify  the  implementation  of  the  organization’s  practices  for  each  instantiation 
and  validate  the  preliminary  findings,  describing  gaps  in  the  implementation  of  model 
practices.  Each  implementation  of  each  practice  is  verified  so  that  it  can  be  compared  to  the 
practices  of  the  People  CMM  model,  and  the  team  characterizes  the  extent  to  which  the 
practices  in  the  model  are  implemented.  Gaps  in  practice  implementation  are  captured  and 
validated  with  members  of  the  organization.  Exemplary  implementations  of  model  practices 
can  be  highlighted  as  strengths  to  be  included  in  appraisal  outputs. 

Entry  Criteria — Objective  evidence  about  the  implementation  of  practices  in  the 
organization  has  been  collected.  Gaps  in  the  implementation  of  model  practices  have  been 
identified,  and  the  team  is  ready  to  characterize  the  extent  to  which  model  practices  (or 
acceptable  alternatives  to  those  practices)  have  been  implemented.  Descriptions  of  practice 
implementation  gaps  at  the  level  of  the  appraised  organization  are  crafted  for  validation. 

Inputs 

•  appraisal  plan,  with  schedule  and  participants  for  data  validation  activities 

•  data  on  practice  implementation  and  strength  and/or  weakness  statements 

•  data  collection  plan,  specifying  any  additional  information  needed 
Activities — The  three  activities  listed  below  are  required  for  this  process. 

2.2.2. 1  Verify  Objective  Evidence 

2.2.2.2  Characterize  Implementation  of  Model  Practices 
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2.2.2.3  Validate  Practice  Implementation  Gaps 

Outputs 

•  updated  appraisal  data 

-  notes 

-  strength/weakness  statements 

-  annotated  worksheets 

•  updated  appraisal  artifacts 

-  preliminary  findings 

-  revised  data  collection  plan 

-  requests  for  additional  data 

Outcome — The  team’s  confidence  in  the  material  that  will  form  the  basis  for  appraisal 
outputs  is  increased  and  the  process  of  transferring  ownership  of  these  results  has  been 
started.  Any  critical  deficiencies  in  the  data  on  hand  have  been  identified  and  actions  to 
resolve  these  issues  have  been  initiated. 

Exit  Criteria— The  team  has  recorded  data  on  the  implementation  of  practices  in  the 
organization  and  characterized  the  extent  to  which  practices  in  the  model  are  implemented.  In 
addition,  strength  and  weakness  statements  have  been  validated  with  members  of  the 
organization  who  provided  appraisal  data. 

Key  Points— This  activity  spans  a  number  of  distinct  events  in  the  appraisal  method  that 
together  accomplish  the  same  goal — ensuring  the  validity  of  the  appraisal  data  and  associated 
outputs.  Managing  the  interaction  with  people  outside  of  the  team  is  a  vitally  important 
process  to  ensure  that  the  results  are  accurate. 

Tools  and  Techniques— Facilitation  techniques  to  guide  the  team  through  difficult  decisions 
are  important  during  this  activity,  just  as  they  are  during  the  rating  activity.  Techniques  to 
enhance  the  credibility  of  the  preliminary  findings  are  also  important.  Using  a  flip  chart  or 
note-taker  during  the  presentation  of  preliminary  findings  is  often  effective  for  instilling 
confidence  among  audience  members. 

Metrics — Planned  versus  actual  effort  expended  for  this  activity  (as  with  all  activities)  will 
assist  in  monitoring  progress  and  planning  subsequent  appraisals.  Gauging  the  level  of 
acceptance  for  preliminary  findings  can  be  facilitated  by  computing  the  percentage  of 
findings  adjusted  based  on  feedback,  then  comparing  this  value  with  past  experience. 

Metrics  are  captured  during  the  appraisal  and  maintained  in  the  appraisal  plan. 

Verification  and  Validation— The  attendees  of  preliminary  findings  presentations  are  likely 

•  to  express  agreement  or  discuss  the  data  being  validated.  The  appraisal  team  leader  needs  to 
ensure  active  participation  in  these  activities  as  a  way  of  verifying  that  the  process  is  working 
as  intended.  The  actions  taken  following  the  appraisal  will  provide  feedback  to  help  validate 
the  success  of  this  activity. 
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Records — Characterizations  of  practice  implementation,  strength  and/or  weakness 
statements,  and  changes  made  based  on  feedback  are  recorded  for  subsequent  use  by  the 
team. 

Tailoring — Validating  data  is  required,  but  a  variety  of  choices  for  orchestrating  this  process 
are  available.  The  most  common  approach  is  the  preliminary  findings  presentation.  The  use 
of  an  instrument  or  a  more  targeted  focus-group  approach  to  validate  statements  of  practice 
implementation  gaps  is  permitted  (e.g.  a  preliminary  report  that  is  reviewed  by 
representatives  in  the  organization  or  the  use  of  a  subset  of  the  participants  interviewed). 

Also,  the  relative  emphasis  of  mini-team-based  verification  and  verification  carried  out  by  the 
team  as  a  whole  can  be  adjusted  to  meet  the  skills  and  preferences  of  the  team  at  hand. 

Interfaces  with  Other  Processes — During  the  conduct  of  an  appraisal,  the  team  must  gather 
and  analyze  a  great  deal  of  detailed  information.  Processes  described  earlier  in  this  document 
clarify  how  data  are  gathered  and  examined.  The  process  described  here  is  focused  on 
understanding  the  information  revealed  by  the  data.  The  processes  described  after  this  one  are 
focused  on  recording  important  information  and  making  reliable  and  valid  rating  judgments 
based  on  the  verified  and  validated  data. 

Summary  of  Activities — The  initial  objective  evidence  provided  by  the  organization  is  used 
to  understand  how  practices  are  to  be  implemented.  Members  of  the  appraisal  team  then  seek 
information  to  confirm  that  the  intended  practices  are  indeed  implemented.  This  first 
validation  activity  (2.2.2. 1  Verify  Objective  Evidence)  might  reveal  gaps  in  the  actual 
implementation  that  are  not  apparent  in  the  initial  objective  evidence  provided  by  the 
organization.  The  next  verification  activity  (2.2.2.2  Characterize  Implementation  of  Model 
Practices)  then  compares  the  implemented  practices  to  the  practices  in  the  People  CMM 
model. 

This  activity  might  also  reveal  gaps  in  the  implementation(s)  that  will  later  affect  the  ratings 
assigned  by  the  team.  Standard  characterizations  to  capture  the  extent  of  practice 
implementation,  first  at  the  unit  level  and  then  at  the  appraised  organization  level,  are 
recorded  by  the  team,  along  with  descriptions  of  gaps  in  implementation.  When  team 
members  have  achieved  their  planned  coverage  of  data  collection,  the  descriptions  of  gaps 
are  validated  with  the  members  of  the  organization.  This  final  activity  before  rating  allows 
team  members  to  build  confidence  that  their  investigation  has  been  thorough,  and  the 
members  of  the  organization  are  provided  with  an  opportunity  to  correct  any  perceived  errors 
in  the  appraisal  data. 

2.2.2.1  Verify  Objective  Evidence 

Activity  Description — The  appraisal  team  must  establish  a  clear  understanding  of  the 
practices  implemented  in  the  organization.  Typically,  the  organization  provides  a  set  of 
objective  evidence  at  the  beginning  of  the  appraisal  process,  and  the  team  sets  out  to  verify 
the  instances  where  those  practices  are  implemented.  For  practices  reflecting  unit-  or 
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workgroup-level  activities,  the  team  must  determine  if  each  selected  unit  or  workgroup  in  the 
appraised  organization  has  evidence  of  implementation.  For  practices  reflecting  organization- 
level  activities,  the  team  must  understand  the  organization-level  implementation  as  well  as 
any  activities  involving  the  unit  or  workgroup  that  indicate  the  implementation  of  the 
practice. 

Required  Practices 

•  Verify  the  appropriateness  of  direct  artifacts  provided  by  each  instantiation  for  practices 
within  the  model  scope  of  the  appraisal. 

•  Verify  the  appropriateness  of  indirect  artifacts  provided  by  each  instantiation  for  practices 
within  the  model  scope  of  the  appraisal. 

•  Verify  the  appropriateness  of  affirmations  provided  by  each  instantiation  for  practices 
within  the  model  scope  of  the  appraisal. 

•  Verify  that  the  implementation  of  each  model  practice  is  supported  by  direct  artifacts  for 
each  instantiation  and  corroborated  by  indirect  artifacts  or  affirmations. 

•  Obtain  face-to-face  affirmations  for  either  at  least  one  instantiation  for  each  model 
practice  in  the  scope  of  the  appraisal,  or  at  least  50%  of  the  practices  corresponding  to 
each  goal  for  each  instantiation.  The  50%  rule  is  applied  to  the  sampled  units  for 
SCAMPI  with  People  CMM.  (Refer  to  Appendix  B  for  more  information  about  the 
50%  rule.) 

•  Generate  statements  describing  gaps  in  the  appraised  organization’s  implemented 
practices  relative  to  practices  defined  in  the  reference  model. 

Parameters  and  Limits — For  practices  implemented  at  the  unit  or  workgroup  level,  direct 
and  indirect  indicators  of  practice  implementation  must  be  examined  for  every  unit  or 
workgroup  sampled  to  represent  the  organization  being  appraised. 

For  practices  implemented  at  the  organization  level,  direct  and  indirect  indicators  of  practice 
implementation  are  examined  in  reference  to  the  appraised  organization  within  the  scope  of 
the  appraisal,  and  not  necessarily  for  each  unit  sampled.  Aspects  of  the  practice  that  are 
implemented  at  the  unit  level  must  be  investigated  for  every  unit  sampled  to  represent  the 
appraised  organization. 

One  or  more  direct  artifacts  are  needed  to  verify  implementation  of  each  model  practice. 
Indirect  indicators  can  include  artifacts  or  affirmations.  Objective  evidence  is  differentiated  in 
terms  of  different  types  of  PEs  (direct  artifacts,  indirect  artifacts,  and  affirmations),  as 
described  in  activity  2. 1.4.5  Obtain  Initial  Objective  Evidence. 

Coverage  criteria  for  face-to-face  affirmations  are  focused  at  the  goal  and  appraised 
•organization  level. 

Optional  Practices— At  the  discretion  of  the  appraisal  team  leader,  verification  of  practices 
at  the  instantiation  level  might  be  carried  out  solely  by  the  mini-teams.  Team-wide  review 
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and  consensus  about  practice  implementation  can  then  focus  on  the  aggregate-level 
characterizations.  Mini-team  consensus  should  be  planned  into  the  schedule  and  performed 
prior  to  the  full  team  consensus  discussions. 

At  the  discretion  of  the  appraisal  team  leader,  the  verification  of  practice  implementation  at 
the  unit  and  workgroup  level  can  be  reviewed  for  consensus  by  the  entire  team.  Each  mini¬ 
team  provides  an  overview  of  practice  implementation  indicators  for  each  unit  and 
workgroup  sampled  to  represent  the  appraised  organization. 

A  mix  of  the  two  strategies  above  can  be  used,  selectively  reviewing  targeted  PAs  in  different 
ways,  or  gradually  changing  from  one  strategy  to  the  other  as  the  team  gains  familiarity  with 
the  data  and  the  process. 

Implementation  Guidance — The  typical  work  products  listed  in  the  People  CMM  model 
provide  examples  of  artifacts  that  can  be  used  as  indicators  of  practice  implementation. 
However,  the  model  does  not  distinguish  between  direct  and  indirect  artifacts;  these  are 
examples  and  are  not  required.  Alternatives  can  be  used  for  both  direct  and  indirect  artifacts. 
(See  Appendix  B.) 

Typically,  much  of  the  objective  evidence  required  to  perform  this  verification  is  provided  in 
advance  of  the  on-site  period.  The  primary  focus  of  data  collection  is  to  permit  the  team  to 
verify  that  the  intended  practices  are  implemented  across  the  appraised  organization.  Where 
the  implemented  practices  differ  from  the  intended  practices,  the  objective  evidence  provided 
at  the  start  of  the  appraisal  process  is  annotated  to  more  accurately  reflect  the  implemented 
process  in  the  organization.  These  annotations  are  typically  statements  describing  a  gap  in  the 
implementation  of  a  model  practice,  some  of  which  will  eventually  become  findings. 

Where  gaps  exist  in  the  objective  evidence  provided  in  advance,  the  appraisal  team  is  forced 
to  undertake  data  collection  activities  to  populate  the  data  set  from  scratch.  An  organization 
that  has  a  substantial  process  improvement  infrastructure  in  place  is  expected  to  have 
documented  its  implementation  of  the  model  in  detail.  For  organizations  with  relatively  little 
experience  using  People  CMM,  the  cost  of  this  discovery  process  might  be  so  great  that 
undertaking  an  ARC  Class  A  appraisal,  such  as  SCAMPI  with  People  CMM,  might  not  be 
cost-effective.  For  such  organizations,  a  Class  B  or  C  appraisal  might  be  more  appropriate  to 
determine  gaps  in  organizational  performance. 

Only  after  team  members  have  a  clear  understanding  of  the  implemented  practices  can  they 
compare  them  to  the  model  to  characterize  the  extent  to  which  the  organization  implements 
the  practices  in  the  model  or  acceptable  alternatives.  Artifacts  that  result  from  the 
performance  of  the  practice  will  be  available  for  viewing  by  the  team.  These  artifacts,  as  well 
as  face-to-face  interactions  with  members  of  the  organization  enacting  the  practice,  help  to 
verify  that  the  practice  was  enacted  as  intended  by  the  maintainers  of  the  organizational 
process. 
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2.2.2.2  Characterize  Implementation  of  Model  Practices 

Activity  Description— Once  a  critical  mass  of  practice  implementation  evidence  has  been 
verified,  the  team  (or  mini-team)  turns  to  characterizing  the  implementation  of  model 
practices.  For  each  practice  in  the  model  included  in  the  selected  scope  and  each  instance  of 
expected  use,  the  team  will  document  a  characterization  of  the  extent  to  which  the  model 
practice  (or  an  acceptable  alternative)  has  been  implemented.  These  unit  and  workgroup-level 
characterizations  are  then  aggregated  to  the  appraised  organization  level. 

Characterizations  of  practice  implementation  are  used  as  a  means  to  focus  appraisal  team 
effort  on  areas  where  professional  judgment  is  needed  and  to  aid  in  reaching  team  consensus 
on  the  extent  to  which  practices  are  implemented. 

Required  Practices 

•  Characterize,  for  each  instantiation,  the  extent  to  which  reference  model  practices  are 
implemented. 

•  Aggregate  practice  implementation  characterization  values  from  the  instantiation  level  to 
the  appraised  organization  level. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — The  following  table  summarizes  rules  for  characterizing 
instantiation-level  implementations  of  practices.  Consensus  of  at  least  a  subset  of  appraisal 
team  members  (e.g.,  mini-team  members)  is  necessary  for  instantiation-level 
characterizations. 


Table  1:  Rules  for  Characterizing  Instantiation-Level  Implementations  of  Practices 


pLabel'' 

'Ho;.  V  '  ; 

Meaning 

Fully  Implemented  (FI) 

•  The  direct  artifact  is  present  and  judged  to  be  appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to  confirm 
the  implementation. 

•  No  substantial  weaknesses  were  noted. 

Largely  Implemented  (LI) 

•  The  direct  artifact  is  present  and  judged  to  be  appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to  confirm 
the  implementation. 

•  One  or  more  weaknesses  were  noted. 

Partially  Implemented  (PI) 

•  The  direct  artifact  is  absent  or  judged  to  be  inadequate. 

•  Artifacts  or  affirmations  suggest  that  some  aspects  of  the  practice 
are  implemented. 

•  One  or  more  weaknesses  were  noted. 

Not  Implemented  (NI) 

Any  situation  not  covered  above. 
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When  tagging  using  the  process  area  workbook  sheets  for  each  PA,  an  artifact  (direct  or 
indirect)  will  only  be  noted  when  it  is  judged  to  be  appropriate  and  adequate.  A  weakness 
might  exist  for  a  particular  unit  or  workgroup  when  a  required  artifact  is  not  in  evidence, 
while  at  the  same  time  other  units  and  workgroups  do  have  the  evidence. 

It  is  possible  that  some  practices  will  not  have  direct  or  indirect  artifacts  in  the  People  CMM. 
Refer  to  Appendix  C  for  more  information. 

Consensus  of  all  members  of  the  appraisal  team  is  necessary  for  appraised  organization-level 
characterizations. 

The  following  table  summarizes  rules  for  aggregating  instantiation-level  characterizations  to 
derive  appraised  organization-level  characterizations.  The  column  labeled  “condition”  is  the 
input  condition — the  practice  implementation  characterizations  for  the  set  of  sampled  units. 
The  column  labeled  “outcome”  is  the  resultant  aggregated  practice  implementation 
characterization  at  the  appraised  organization  level. 

Table  2:  Rules  for  Aggregating  Instantiation-Level  Characterizations  to  Derive 
Appraised  Organization-Level  Characterizations 


.Condition 

Outcome 

Remarks 

All  X  (e.g.,  all  LI) 

X 

All  instantiations  have  the  same  characterization. 

All  LI  or  H 

LI 

All  instantiations  are  characterized  LI  or  higher. 

Any  PI,  no  NI 

LI  or  PI 

Team  judgment  is  applied  to  choose  LI  or  PI  for  the 
organizational  unit. 

Any  NI 

NI,  PI,  or  LI 

Team  judgment  is  applied  to  choose  LI  or  PI  for  the 
organizational  unit. 

Optional  Practices — While  the  initial  characterization  of  practice  implementation  might  be 
proposed  by  a  mini-team  or  some  subset  of  the  team,  the  selections  listed  below  are  available. 

•  Instantiation-level  characterization  of  practice  implementation  can  be  reviewed  by  the 
entire  team  for  consensus. 

•  Team-wide  review  and  consensus  on  practice  implementation  characterization  can  be 
reserved  for  the  appraised  organization  level. 

•  A  mix  of  the  two  strategies  above,  tailored  to  match  the  learning  curve  of  the  team 
members  or  to  reflect  the  prioritization  of  particular  PAs,  can  be  used. 

Additionally  the  Lead  Appraiser  reviews  these  tables  with  mini-teams  during  the  appraisal. 

Implementation  Guidance — When  the  team  is  ready  to  perform  the  ratings,  these 
characterizations  simplify  the  judgments.  The  team  is  then  able  to  focus  on  the  aggregation  of 
weaknesses  observed  to  determine  the  goal  satisfaction  ratings  (as  explained  in  process  2.2.4 
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Generate  Appraisal  Results).  Situations  where  the  unit  has  not  yet  reached  the  time  where  the 
practice  would  be  enacted  are  omitted  from  this  characterization.  The  appraisal-planning 
activities  are  expected  to  prevent  situations  that  severely  limit  the  examples  of  actual 
implementation  for  any  given  practice.  These  are  rare  in  the  People  CMM  where,  as  an 
organization-centric  model,  it  is  expected  that  organization  practices  would  have  been 
deployed  across  units  and  workgroups  in  the  organization.  Nonetheless,  rare  situations  might 
exist  due  to  timing  of  the  deployment  activities  if  full  organization  roll  out  might  not  have 
occurred  completely. 

The  characterization  of  People  CMM  practice  implementation  begins  as  soon  as  sufficient 
data  are  available.  It  is  not  necessary  that  data  for  every  instantiation  be  available  before  the 
implementation  of  any  given  practice  can  be  characterized  at  the  instantiation  level.  However, 
before  the  implementation  of  a  practice  across  the  appraised  organization  can  be 
characterized,  the  instantiation-level  characterizations  must  be  completed. 

Each  instance  of  practice  enactment  is  characterized  using  the  instantiation  level 
characterization  scheme.  The  characterization  of  practice  implementation  for  the  appraised 
organization  is  carried  out  using  the  aggregation  rules  summarized  in  Table  2.  These  rules 
provide  a  basis  for  identifying  the  areas  where  professional  judgment  is  required  and  simplify 
the  areas  where  the  data  are  unanimous. 

2.2.2.3  Validate  Practice  Implementation  Gaps 

Activity  Description-Verification  activities  lead  to  statements  summarizing  gaps 
(weaknesses)  in  the  implementation  of  model  practices.  Optionally,  statements  reflecting 
exceptional  implementations  of  model  practices  (strengths)  might  also  be  generated.  These 
statements  can  be  generated  at  various  points  in  the  appraisal  process,  such  as  when  initial 
objective  evidence  is  obtained.  Implemented  practices  are  compared  to  the  practices  in  the 
reference  model,  and  the  extent  of  implementation  is  characterized  for  each  unit  or  for  the 
appraised  organization. 

To  prepare  for  validating  this  information,  the  appraisal  team  generates  preliminary  findings 
that  summarize  the  practice  implementation  gaps.  The  preliminary  findings  are  written  in 
reference  to  a  single  model  practice  and  are  abstracted  to  the  level  of  the  appraised 
organization.  The  statements  should  not  reference  a  specific  individual,  unit,  workgroup,  or 
other  identifiable  organizational  sub-unit. 

This  is  still  primarily  a  data  collection  activity,  and  the  intent  is  to  validate  the  appraisal 
team’s  understanding  of  the  processes  implemented  within  the  appraised  organization. 
Feedback  from  participants  might  result  in  modifications  to  the  appraisal  team’s  inventory  of 
objective  evidence.  The  results  of  the  validation  activity  must  be  considered  in  the 
formulation  of  final  findings  and  goal  ratings.  These  latter  activities  cannot  commence  until 
after  the  validation  activity  has  occurred. 
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Required  Practices 

•  Validate  preliminary  findings  with  member(s)  of  the  legal  community  for  the 
organization. 

•  Generate  preliminary  findings  summarizing  gaps  in  practice  implementation  observed 
with  the  appraised  organization  relative  to  reference  model  practices. 

•  Validate  preliminary  findings  with  members  of  the  appraised  organization. 

All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — Full  appraisal  team  consensus  must  be  reached  on  the  preliminary 
findings  before  they  are  provided  to  the  appraised  organization  for  validation. 

Preliminary  findings  must  be  corroborated  via  multiple  practice  implementation  indicator 
types  (direct,  indirect,  affirmation).  The  appraisal  team  should  request  additional  information 
in  areas  where  the  inventory  of  objective  evidence  is  insufficient.  Preliminary  findings  must 
not  refer  to  specific  individuals,  units,  or  organizational  sub-units. 

Every  model  practice  characterized  as  either  not  implemented  or  partially  implemented,  at  the 
appraised  organization  level,  must  have  at  least  one  preliminary  finding  associated  with  it. 

All  practices  will  have  observations  which  can  become  findings,  whether  a  strength  or 
weakness.  Significant  strengths  will  later  be  summarized  by  goal  (i.e.,  the  goal  will  be  stated 
directly  from  People  CMM  for  the  PA,  and  findings  will  include  a  bulleted  summary  of 
significant  strengths). 

At  least  one  representative  from  each  unit  and  from  any  associated  staff  functions  must 
participate  in  the  set  of  validation  activities.  Only  appraisal  participants  may  participate  (i.e., 
only  people  who  provided  data  may  participate  in  validation). 

Legal  representatives  for  the  organization  must  review  the  draft  findings  to  ensure  there  is  no 
breach  of  confidentiality  or  potential  issues  between  management  and  the  workforce. 

A  minimum  of  one  validation  session  is  required,  and  no  more  than  five  are  recommended — 
although  no  maximum  limit  is  specified.  Three  sessions  is  most  common.  The  rules  of 
confidentiality  and  the  expected  use  of  appraisal  data  must  be  communicated  to  participants 
in  each  validation  activity. 

Optional  Practices — Preliminary  findings  (and  other  appraisal  results)  focused  on  specific 
projects,  divisions,  or  other  organizational  sub-units  might  be  generated  if  they  are  reflected 
in  the  appraisal  objectives  and  constraints.  This  tailoring  option  also  requires  that  the 
members  of  the  organization  participating  in  the  appraisal  be  fully  informed  of  the  intended 
use  of  the  information  they  provide  to  the  appraisal  team. 

Activities  performed  during  this  task  ensure  that  the  final  findings  presentation  does  not  have 
unrealized  legal  implications.  The  appraisal  team  reviews  the  preliminary  findings  briefing 
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with  the  legal  counsel  or  a  representative  of  the  legal  unit  of  the  assessed  organization.  This 
review  ensures  that  the  appraisal  team  does  not  present  material  that  would  adversely  affect 
legal  actions  or  grievances  pending  in  the  organization.  The  intent  of  this  review  is  not  to 
have  the  legal  unit  sanitize  the  findings  or  alter  them  in  any  arbitrary  way.  However,  since  the 
findings  represent  documented  information  that  could  be  brought  into  future  legal 
proceedings  (discoverable  information),  the  legal  unit  should  be  aware  of  them  and  should 
have  the  opportunity  to  recommend  wording.  If  needed,  the  appraisal  team  will  revise  the 
preliminary  findings  briefing  before  presenting  it  to  process  owners.  Otherwise,  changes 
requested  by  the  legal  unit  are  made  in  preparing  the  final  findings  briefing. 

In  addition  to  the  preliminary  findings,  the  legal  unit  should  be  briefed  on  the  confidentiality 
aspects  of  the  appraisal  process,  including  the  confidentiality  agreements  that  have  been 
signed  by  all  appraisal  team  members.  Participants  at  this  review  should  be  limited  to  the 
relevant  legal  counsel  or  legal  department  staff  and  the  appraisal  team.  The  appraisal  sponsor 
should  not  attend  this  review  session. 

Some  of  the  activities  typically  performed  during  this  task  are  listed  below. 

•  Review  preliminary  findings  briefing. 

•  Record  recommended  changes  and  revise  preliminary  findings  briefing,  if  necessary. 

Implementation  Guidance — Preliminary  findings  are  the  building  blocks  that  lead  to  the 
judgment  of  goal  satisfaction  and  form  the  basis  for  the  final  findings.  As  an  intermediate 
artifact  of  the  appraisal  process,  preliminary  findings  are  used  to  ensure  traceability  between 
appraisal  inputs  and  appraisal  outputs. 

Feedback  from  participants  about  the  preliminary  findings  should  be  solicited  by  the 
appraisal  team  and  considered  for  possible  revisions  to  its  inventory  of  objective  evidence. 

It  is  not  expected  that  preliminary  findings  will  provide  a  detailed  listing  of  the 
implementation  status  of  every  model  practice  in  every  sampled  unit. 

Furthermore,  it  is  not  expected  that  the  preliminary  findings  will  identify  the  status  of 
individual  units  and  workgroups  with  regard  to  practice  implementation  or  goal  achievement. 
An  appraisal  sponsor  might  request  these  more  detailed  appraisal  results.  The  appraisal  team 
leader  should  negotiate  for  the  proper  allocation  of  time  to  accommodate  this  tailoring  option, 
and  the  expectation  that  such  information  will  be  preserved  at  the  end  of  the  appraisal  should 
be  made  clear  to  all  appraisal  participants. 

Practice  feedback  will  be  provided,  with  strengths  summarized  at  the  goal  level  in  a  bulleted 
list.  Weaknesses  are  written  as  separate  findings  for  each  practice  within  the  goal  that  has  a 
noted  weakness. 

Implementation  Guidance:  Preliminary  Findings  Presentations — An  interactive 
presentation  is  the  most  effective  mechanism  for  validating  the  preliminary  findings.  The 
members  of  the  organization  who  provided  data  to  the  appraisal  team  are  typically  brought 
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together  in  a  conference  room,  and  a  slide  presentation  is  used  to  review  the  preliminary 
findings  in  an  effort  to  invite  people  to  provide  additional  data  or  express  their  agreement 
with  the  summary  statements.  The  audience  is  often  grouped  by  seniority  in  the  organization, 
and  separate  presentations  are  made  for  workforce,  process  owners,  and  managers. 

During  the  presentation,  one  or  more  members  of  the  team  review  the  preliminary  findings 
statements  and  provide  the  audience  with  an  opportunity  to  comment  or  ask  questions.  The 
presenter  uses  only  the  words  crafted  by  the  appraisal  team  and  avoids  elaborating  on  the 
findings  using  his  or  her  own  words.  When  questions  are  asked  about  a  preliminary  finding, 
the  team  leader  provides  any  clarification  needed.  However,  team  members  should  avoid  the 
appearance  that  they  are  justifying  the  content  of  the  statement. 

The  detailed  data  that  led  to  the  preliminary  findings  must  be  protected,  and  there  is  no 
negotiation  for  rewording  or  eliminating  findings.  The  appraisal  team  must  record  new  data 
made  available  to  them  without  commenting  on  how  the  data  might  be  interpreted  or  how  the 
findings  might  need  to  change. 

Implementation  Guidance:  Survey  Instrument — Finally,  a  survey  instrument  can  be  used 
in  addition  (or  as  an  alternative)  to  the  techniques  above.  A  carefully  worded  instrument  that 
asks  respondents  to  rate  their  level  of  agreement  with  the  finding  statement  and  provides  an 
opportunity  for  written  feedback  can  provide  a  low-cost  and  timely  source  of  data  for  the 
team. 


2.2.3  Document  Objective  Evidence 

Purpose — Transforming  the  data  into  lasting  records  that  document  practice  implementation 
and  strengths  and  weaknesses. 

Entry  Criteria — Planning  activities  for  the  appraisal  are  complete,  including  the  selection 
and  preparation  of  the  appraisal  team.  At  least  one  data  collection  activity  has  been 
conducted,  and  appraisal-relevant  data  are  available  to  record. 

Inputs — Appraisal  data,  including  the  following: 

•  notes  taken  during  data  collection  activities  (if  applicable) 

•  annotated  worksheets  or  other  work  aids  containing  data  (if  applicable) 

•  strengths  and  weaknesses  documented  from  previous  activities 

•  data  collection  plan 

Activities — The  four  activities  listed  below  are  required  for  this  process. 

2.2.3. 1  Take/Review/Tag  Notes 

2.2.3.2  Record  Presence/Absence  of  Objective  Evidence 

2.2.3.3  Document  Practice  Implementation  Gaps 

2.2.3.4  Review  and  Update  the  Data  Collection  Plan 
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Outputs 

•  updated  appraisal  data,  including  the  following: 

-  noted  practice  implementation  gaps  (if  any) 

-  revised  data  collection  plan  (if  applicable) 

-  annotated  worksheets 

•  requests  for  additional  data  (interviewees  or  documents) 

Outcome — Individual  team  members  understand  the  data  collected  thus  far  and  have 
information  to  guide  any  needed  subsequent  data  collection. 

Exit  Criteria — All  data  from  the  most  recent  data  collection  session  has  been  captured  as  a 
new  baseline  of  practice  implementation  evidence  or  strength  and  weakness  statements.  The 
data-gathering  plans  have  been  updated  to  reflect  the  need  for  additional  information  and 
topics  that  can  be  removed  from  active  investigation. 

Key  Points— This  process  has  traditionally  been  the  most  difficult  one  to  manage  during  an 
appraisal.  Members  of  the  team  will  vary  a  great  deal  in  their  productivity  and  style  of  work. 
The  team  leader  must  be  very  attentive  to  the  progress  of  each  team  member  and  take 
effective  corrective  actions  to  ensure  team  progress. 

Tools  and  Techniques— Because  of  the  challenging  nature  of  this  activity,  Lead  Appraisers 
often  have  strong  preferences  for  tools  and  techniques  they  have  found  to  be  successful. 

Some  high-level  tools  and  techniques  that  have  been  effective  for  others  effective  include  the 
following: 

•  work  aids  such  as  wall  charts,  spreadsheet  programs,  and  automated  database  tools  to 
help  track  the  status  of  data  collection 

•  mini-teams,  where  pairs  (or  triplets)  of  team  members  are  assigned  specific  PAs 

•  explicit  review  of  the  effort  spent,  in  real  time.  Time  management  is  a  critical  skill  for 
this  activity. 

•  a  variety  of  techniques  for  structuring  team  notebooks  and  formats  for  recording  notes 

•  team  norms  regarding  techniques  for  managing  debates  and  divergent  views.  (Make 
explicit  well  in  advance.) 

Metrics — As  mentioned  above,  tracking  the  effort  expended  during  this  activity  (in  real  time) 
is  a  valuable  technique.  The  ability  to  learn  quickly  the  rate  at  which  each  team  member 
works  is  a  skill  that  experienced  Lead  Appraisers  develop  using  effort  and  duration  metrics. 

Metrics  are  captured  during  the  appraisal. 

•  Verification  and  Validation — The  method  rules  for  recording  traceability  and  validating 
data  contribute  significantly  to  the  verification  and  validation  of  the  appraisal  data.  The  role 
of  the  appraisal  team  leader  in  monitoring  progress  and  the  consensus  decision  making 
process  also  serve  as  important  verification  and  validation  activities. 
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Records — All  appraisal  data  are  recorded  with  full  traceability  to  information  sources  and 
the  model  components  to  which  they  pertain.  The  full  detail  in  this  traceability  contains 
sensitive  information  that  should  not  be  provided  to  people  outside  of  the  appraisal  team. 

Data  must  never  be  attributed  to  individuals  or  groups  even  if  some  of  the  detailed  data  are 
provided  to  the  engineering  process  group  for  use  in  process  improvement. 

All  appraisal  data  are  recorded  in  the  process  area  workbook. 

Tailoring — The  use  of  a  specialized  appraisal  data  management  tool  is  common  in  this 
activity. 

Interfaces  with  Other  Processes — The  mechanics  associated  with  recording  and 
transcribing  objective  evidence  are  described  in  this  section.  There  are  many  links  between 
these  mechanics  and  the  data  collection  process,  as  well  as  the  data  verification  and 
validation  process.  It  is  important  to  understand  that  the  data-recording  process  must  support 
these  other  processes  and  that  the  tools  used  during  an  appraisal  will  need  to  accommodate 
these  linkages.  Typically,  an  integrated  database  tool  is  used  to  manage  all  appraisal  data  that 
result  from  the  analysis  of  notes  taken  during  data  collection. 

Summary  of  Activities— The  notes  taken  by  individual  team  members  provide  the  most 
basic  representation  of  appraisal  data.  These  notes  are  reviewed  and  tagged  or  otherwise 
processed  before  their  information  content  is  transformed  into  lasting  representations.  The 
presence,  absence,  and/or  appropriateness  of  objective  evidence  is  judged  and  recorded  based 
on  the  data  collected.  The  scheme  by  which  this  set  of  records  is  produced  is  an  important 
implementation  choice  made  by  the  appraisal  team  leader  and  must  be  well  understood  by  the 
team.  Gaps  in  the  implemented  practices  are  also  recorded  in  a  consistent  manner  to  ensure 
traceability.  Finally,  the  data  collection  plan  must  be  reviewed  in  light  of  changes  in  the  set  of 
data  available  to  the  team  and  the  remaining  data  needed  to  support  reliable  rating  judgments. 

2.2.3. 1  Take/Review/Tag  Notes 

Activity  Description — As  team  members  examine  data  sources,  they  document  what  the 
objective  evidence  is  (referencing  documents,  presentations,  instruments,  and  interviewee 
comments),  as  well  as  why  or  how  the  objective  evidence  meets  the  intent  of  the  model 
practice. 

In  some  cases,  team  members  record  data  directly  in  the  objective  evidence  tracking  tool 
instead  of  taking  notes  on  paper  or  in  their  notebooks  that  describe  the  objective  evidence. 

For  all  interviews  and  presentations,  the  team  members  must  take  notes  that  capture  the 
objective  evidence  before  they  move  to  the  annotation  of  the  objective  evidence  tracking  tool. 

Required  Practices 

•  Record  notes  obtained  from  objective  evidence  data-gathering  sessions. 

•  Relate  notes  to  corresponding  practices  in  the  appraisal  reference  model. 
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All  required  practices  performance  decisions  are  captured  in  the  appraisal  plan. 

Parameters  and  Limits — Every  team  member  present  must  take  notes  during  interviews  and 
presentations.  These  notes  must  cover  all  areas  investigated  during  the  interview  and  are  not 
limited  to  the  PAs  assigned  to  the  individual  team  member  (i.e.,  everybody  takes  notes  on 
everything). 

During  document  reviews  and  the  review  of  instruments,  notes  must  be  taken  to  preserve 
specific  context  or  focused  references  if  the  rationale  for  accepting  the  objective  evidence  is 
not  self-evident. 

Whenever  notes  are  taken  in  a  data-gathering  session,  individual  team  members  should 
review  their  notes  immediately  after  the  conclusion  of  the  session  and  tag  significant  items 
that  relate  to  one  or  more  model  practices.  This  review  and  tagging  process  must  occur  within 
24  hours  of  the  data-gathering  session. 

Optional  Practices — Tagging  schemes  that  show  traceability  to  model  practices  and 
techniques  for  highlighting  phrases  are  determined  by  the  preferences  of  the  note  taker.  A 
variety  of  formats  has  been  devised  for  team  member  notebooks  to  facilitate  note  taking  and 
tracking  raw  data  during  appraisals.  Frequently,  the  questions  used  during  an  interview  are 
printed  and  collated  in  a  team  member  notebook  that  contains  note-taking  forms  and  other 
useful  information  such  as  interview  schedules  and  document  lists. 

Notes  can  be  recorded  for  items  that  have  significant  positive  or  negative  impact  on  the 
enactment  of  processes  within  the  appraised  organization,  even  if  they  are  not  directly  related 
to  model  practices.  These  might  ultimately  be  reflected  in  non-model  findings  reported  to  the 
appraised  organization. 

Implementation  Guidance — The  raw  notes  taken  during  an  appraisal  are  treated  as 
confidential  information  and  may  not  be  provided  to  any  person  outside  of  the  appraisal 
team.  Team  members  are  required  to  destroy  their  notes  in  a  secure  manner  at  the  conclusion 
of  the  appraisal.  This  ensures  that  the  attribution  of  detailed  information  to  individuals  in  the 
organization  cannot  lead  to  inappropriate  consequences  following  the  appraisal. 

Implementation  Guidance;  Taking  Notes— Team  members  actively  take  notes  during  all 
data-gathering  sessions.  The  purpose  is  to  record  verbatim  what  the  information  source 
reveals  about  the  implementation  of  practices  in  the  unit  or  organization.  Note-taking  is  done 
for  all  types  of  objective  evidence. 

The  analysis  of  instruments  yields  information  and  references  regarding  the  implementation 
of  practices,  ideally  with  traceability  to  the  model.  While  reviewing  documents,  note  specific 
phrases  or  references,  and  record  the  document  name  and  page  number.  When  receiving 
presentations,  phrases  or  references  provided  as  elaboration  are  captured  in  notes.  Interviews 
are  the  most  intensive  activity.  The  purpose  is  to  record  what  the  interviewees  said,  not  what 
the  team  member  believes  they  meant. 
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Implementation  Guidance:  Reviewing  Notes — The  context  in  which  the  data  are 
provided — be  it  during  an  interview,  presentation,  or  in  a  document — affects  the  proper 
interpretation  of  the  facts.  For  example,  notes  taken  during  an  interview  are  based  on  a  give 
and  take  between  the  interviewer  and  the  interviewee.  The  threads  of  discussion  often 
provide  a  context  that  might  not  be  reflected  in  a  single  excerpt  from  the  middle  of  the 
interview.  Note  takers  should  review  their  work  to  ensure  that  such  contextual  information 
can  be  preserved  in  their  recollection,  and  preferably  in  the  annotation  of  their  notes. 

Implementation  Guidance:  Tagging  Notes — As  notes  are  reviewed,  team  members  often 
use  highlighter  pens  or  annotation  schemes  to  identify  the  most  salient  excerpts.  The  PA 
and/or  practice  to  which  the  information  applies  might  be  written  in  colored  ink  over  the  raw 
notes.  All  notes  should  identify  the  data-gathering  session,  and  the  pages  should  be  numbered 
to  preserve  the  sequence  of  information.  For  notes  taken  during  interviews,  it  is  often  very 
useful  to  draw  a  seating  chart  to  show  where  each  person  was  sitting  during  the  interview. 
Scripts  prepared  in  advance  of  scheduled  interviews  can  be  tagged  to  help  relate  responses  to 
appropriate  sections  of  the  reference  model.  Some  interviewee  responses  might  deal  with 
model  practices  other  than  those  targeted  by  a  given  question,  which  would  necessitate 
additional  tagging. 

2.2.3.2  Record  Presence/Absence  of  Objective  Evidence 

Activity  Description — The  presence  or  absence  of  appropriate  objective  evidence  for  each 
model  practice  in  the  scope  of  the  appraisal  is  determined  based  on  information  obtained 
from  data-gathering  sessions.  Annotations  are  made  to  indicate  the  source,  relevance,  and 
coverage  of  objective  evidence  collected.  If  the  data  source  does  not  make  it  obvious  why  the 
objective  evidence  is  appropriate,  a  comment  can  be  added  to  the  annotation.  For  example, 
when  an  alternative  training  approach  is  used,  it  might  be  necessary  to  document  why  that 
alternative  meets  the  intent  of  the  model  practice.  Adding  comments  to  the  annotations  can 
help  avoid  rehashing  the  rationale  for  accepting  the  objective  evidence  during  team 
discussions. 

Required  Practices — Record  the  presence  or  absence  of  appropriate  objective  evidence 
collected  for  each  reference  model  practice. 

The  availability  of  objective  evidence  is  captured  in  the  process  area  workbook  tracking  tool. 

Parameters  and  Limits — The  inventory  of  objective  evidence  (in  electronic  or  paper  form) 
is  updated  to  reflect  what  the  data  imply  about  the  implementation  of  particular  practices. 
Throughout  the  appraisal  conduct,  annotations  indicating  the  presence  or  absence  of  objective 
evidence  are  made  for  every  practice  within  the  model  scope  of  the  appraisal.  The  annotation 
scheme  used  must  ensure  that  the  record  reveals  the  following  information: 

•  the  unit  or  appraised  organization  to  which  the  data  apply 

•  the  practice  to  which  the  data  apply 

•  the  type  of  objective  evidence  being  recorded  (i.e.,  direct,  indirect,  or  affirmation) 
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•  whether  the  data  imply  the  presence  or  absence  of  the  objective  evidence 

•  whether  the  data  suggest  that  the  objective  evidence  is  appropriate 

•  comments  about  the  appropriateness  of  the  evidence  (if  needed) 

•  whether  or  not  additional  information  is  needed  before  the  team  can  characterize  the 
extent  to  which  the  practice  is  implemented 

•  a  description  of  what  the  evidence  is,  if  such  a  description  was  not  provided  by  the 
organization  in  advance 

All  evidence  is  captured  in  the  process  area  workbook. 

Optional  Practices — Following  each  verification  session  where  the  presence  or  absence  of 
objective  evidence  is  recorded,  the  team  reviews  the  judgments  about  each  new  piece  of 
objective  evidence.  This  can  be  useful  in  establishing  a  common  understanding  of  the 
expectations  for  objective  evidence,  especially  early  in  the  appraisal. 

Implementation  Guidance— This  activity  represents  the  mechanical  aspects  of  processing 
appraisal  data  and  is  strongly  tied  to  the  activities  described  in  process  2.2.2  Verify  and 
Validate  Objective  Evidence.  It  emphasizes  the  steps  needed  to  update  the  inventory  of 
objective  evidence  and  maintain  traceability  to  data  sources,  while  process  2.2.2  emphasizes 
the  interpretation  of  the  data  collected  and  the  sufficiency  of  objective  evidence  relative  to 
the  appraisal  reference  model. 

Team  members  typically  record  the  presence  or  absence  of  appropriate  objective  evidence 
with  tools  such  as  tracking  tables  or  data  consolidation  worksheets.  Prior  to  the  assignment  of 
goal  ratings,  the  entire  team  reviews  the  status  of  the  objective  evidence  as  reflected  in  the 
annotations  made  by  each  team  member. 

The  data  gathered  during  every  data  collection  session  should  be  related  to  the  practices  in 
use  in  a  unit  or  across  the  organization.  In  recording  the  presence  or  absence  of  objective 
evidence,  the  intent  is  to  inventory  quickly  the  composite  of  factual  information.  Elaboration 
about  the  meaning  of  data  or  how  they  relate  to  other  important  issues  is  captured  either  in 
notes  or  in  the  descriptions  of  practice  implementation  gaps  crafted  by  team  members. 

2.2.3.3  Document  Practice  Implementation  Gaps 

Activity  Description — The  primary  intent  of  this  activity  is  to  derive  summary  prose 
statements  that  describe  the  gap  between  the  objective  evidence  and  what  the  team  was 
looking  for  to  support  a  claim  that  the  model  practice  was  implemented.  The  statements 
explain  why  the  practice  is  not  considered  to  be  fully  implemented.  Statements  of  practice 
implementation  gaps  will  be  validated  with  the  appraised  organization  at  a  later  time. 

Strengths  are  not  recorded  pro  forma  when  practices  are  found  to  be  fully  implemented. 
Where  practices  represent  exemplary  or  significant  implementations  of  the  model  practices, 
the  appraisal  team  highlights  these  as  part  of  the  appraisal  output.  However,  the  primary 
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focus  of  this  benchmarking  method  is  to  help  the  organization  verify  the  implementation  of 
the  model  and  identify  areas  where  work  is  needed. 

All  practices  with  exemplary  or  significant  strengths  or  a  weakness  will  have  a  finding. 
Findings  will  later  be  summarized  by  goal  (i.e.,  the  goal  will  be  stated  directly  from  the 
People  CMM  for  the  PA,  and  bulleted  statements  summarizing  the  practice  strengths  or 
weaknesses  will  be  written  as  a  finding). 

Required  Practices — Describe  in  writing  the  gaps  in  the  appraised  organization’s 
implemented  processes  relative  to  reference  model  practices. 

Any  gaps  are  captured  in  the  process  area  workbook  tracking  tool. 

Parameters  and  Limits — For  any  practice  that  is  not  characterized  as  fully  implemented, 
there  must  be  a  prose  statement  explaining  the  gap  between  what  the  organization  does  and 
what  the  model  expects. 

Regardless  of  the  medium  used,  statements  describing  practice  implementation  gaps  should 
be  annotated  with  the  following  identifying  information: 

•  the  model  component  to  which  the  statement  relates  (i.e.,  PA,  goal,  or  practice) 

•  the  data  collection  session(s)  in  which  the  information  was  uncovered 

•  the  process  instantiation(s)  to  which  the  statement  applies.  Prose  statements  of  practice 
implementation  gaps  presented  to  the  appraised  organization  in  the  form  of  preliminary 
findings  for  validation  must  be  free  of  references  to  specific  individuals  or  units. 

Optional  Practices — Document  strengths  in  the  implementation  of  model  practices  when  the 
team  discovers  exemplary  or  significant  implementations.  Label  implementation  gaps  as 
“opportunities  for  improvement”  to  avoid  the  potentially  negative  connotations  of  labeling 
them  as  weaknesses. 

Document  any  significant  issues  impeding  performance  in  the  organization  even  if  they  do 
not  map  to  the  People  CMM  model.  This  must  be  done  cautiously,  and  the  number  of  these 
issues  should  not  be  greater  than  the  number  of  model-related  issues  reported  by  the  team. 

Implementation  Guidance — The  database  used  to  record  the  inventory  of  objective 
evidence  may  also  be  used  to  record  practice  implementation  gaps  and  strengths,  or  a 
separate  location  or  tool  may  be  used.  Gaps  in  practice  implementation  should  be  recorded  at 
the  level  of  a  particular  instance  of  a  model  practice.  These  precursors  to  preliminary  findings 
are  more  detailed  and  pointed,  while  all  information  presented  outside  of  the  team  is 
aggregated  to  the  goal  and  appraised-organization-level  of  abstraction. 

Try  to  write  observations  as  closely  as  possible  to  what  a  final  finding  would  look  like.  This 
is  good  practice,  but  not  a  method  requirement. 
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Strengths  are  only  documented  if  the  implementation  of  a  practice  is  exceptional  or 
significant  and  reflects  a  strong  asset  in  the  process  in  use.  An  adequate  implementation  of  a 
model  practice  is  not  necessarily  a  strength.  Team  members  should  use  their  collective 
experience  and  judgment  to  determine  whether  or  not  they  have  uncovered  an  exemplary  to 
highlight  in  the  appraisal  output.  Gaps  in  practice  implementation  are  documented  if  the 
objective  evidence  indicates  a  missing  component  in  the  process  or  an  inappropriate  practice 
in  light  of  the  value  the  practice  is  expected  to  add  to  the  achievement  of  the  goal.  That  is, 
practices  that  fail  to  help  the  organization  meet  the  People  CMM  goal  to  which  they  relate 
should  have  a  gap  documented  that  explains  why  the  goal  is  not  met. 

All  practices  might  have  a  finding,  whether  a  strength  or  weakness.  Findings  will  later  be 
summarized  by  goal  (i.e.,  the  goal  will  be  stated  directly  from  People  CMM  for  the  PA,  and 
bulleted  statements  summarizing  the  practice  strengths  and  weaknesses  will  be  written  as  a 
finding). 

2.2.3.4  Review  and  Update  the  Data  Collection  Plan 

Activity  Description — This  activity  is  used  to  continuously  monitor  the  state  of  available 
objective  evidence  and  to  select  the  next  tactic  in  the  pursuit  of  obtaining  full  coverage  of  the 
model  and  the  appraised  organization. 

Required  Practices 

•  Review  the  objective  evidence  collected  and  the  data  collection  plan  to  determine  what 
additional  objective  evidence  is  still  needed  for  sufficient  coverage  of  the  model  scope. 

•  Revise  the  data  collection  plan  to  obtain  additional  objective  evidence  if  data  is  not 
sufficient  to  judge  the  implementation  of  reference  model  practices. 

•  Identify  priorities  for  the  upcoming  data  collection  events  and  reevaluate  the  feasibility  of 
the  schedule  in  light  of  the  current  state  of  the  objective  evidence. 

The  availability  of  any  additional  required  objective  evidence  is  captured  in  the  process  area 
workbook  tracking  tool. 

Parameters  and  Limits— This  activity  must  be  enacted  at  least  once  a  day,  and  a 
consolidated  summary  of  the  appraisal  data  collection  status  must  be  available  to  the  team  at 
the  start  of  each  day  during  which  data  collection  events  are  planned. 

Optional  Practices — In  addition  to  the  daily  status  mentioned  above,  more  frequent  status 
checks  may  be  conducted.  These  interim  status  checks  are  not  aggregated  for  a  team-wide 
view  of  status  unless  the  appraisal  team  leader  finds  that  beneficial. 

Implementation  Guidance— The  data  collection  status  summarizes  the  differences  between 
the  objective  evidence  on  hand  and  the  evidence  needed  to  support  the  creation  of  appraisal 
outputs  (e.g.,  ratings).  Annotations  regarding  the  presence  and  appropriateness  of  objective 
evidence  allow  the  team  to  inventory  the  state  of  the  knowledge  base.  This  status  then  drives 
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requirements  for  the  collection  of  more  data,  which  must  be  met  by  the  data  collection  plan. 
The  annotation  of  the  inventory  of  objective  evidence  is  described  in  process  2.2.2  Verify  and 
Validate  Objective  Evidence. 

The  plan  for  future  data  collection  should  be  revisited  and  updated  as  necessary.  There  might 
be  several  situations  in  which  additional  data  are  required  for  the  team  to  sufficiently 
characterize  the  implementation  of  reference  model  practices.  Some  examples  of  these 
situations  are  listed  below. 

•  The  process  of  reconciling  new  data  with  old  might  identify  conflicts  or  ambiguities  in 
the  data  that  require  clarification. 

•  The  search  for  objective  evidence  from  previous  appraisals  or  mini-appraisals  might  lead 
to  the  discovery  of  one  or  more  previously  undocumented  practice(s)  in  the  organization. 

•  Attempts  to  confirm  the  use  of  a  particular  practice  or  tool  in  a  unit  might  have  been 
unsuccessful. 

Prioritizing  data  needs  and  allocating  data  collection  effort  are  ongoing  activities  that  the 
appraisal  team  leader  oversees.  The  data  collection  status  summary  might  be  performed  by 
the  appraisal  team  leader  and  reported  to  the  team  members,  or  the  appraisal  team  leader 
might  elect  to  have  each  mini-team  perform  this  activity  for  its  assigned  PAs. 

Specific  information  needed  to  resolve  ambiguities  or  conflicts  in  the  existing  data  should  be 
documented  for  follow-up  by  one  or  more  members  of  the  team.  For  detailed  data  items  that 
have  a  limited  scope  of  impact,  the  notes  of  individual  team  members  might  be  adequate  to 
document  the  data  needed.  For  example,  whether  or  not  a  particular  person  is  involved  in  a 
meeting  or  reviews  a  given  document  can  be  confirmed  by  a  question  asked  during  an  on-call 
interview.  Therefore,  a  note  made  by  an  individual  team  member  to  make  sure  the  question  is 
asked  might  suffice.  In  contrast,  if  conflicting  information  is  uncovered  about  whether  or  not 
a  given  event  occurred  (such  as  a  meeting)  more  visibility  for  this  conflict  might  be  needed 
among  the  team  members  to  understand  why  the  information  collected  thus  far  is  not 
consistent.  In  such  a  case,  the  person(s)  responsible  for  the  PA  where  that  practice  resides 
might  need  to  alert  the  team  to  the  conflicting  data  and  facilitate  a  team  discussion  to  seek 
clarity  and  additional  data.  This  could  lead  to  the  crafting  of  a  specific  interview  question  to 
be  used  in  standard  interviews. 

The  data  collection  plan  and  inventory  of  objective  evidence  allows  the  appraisal  team  to 
continuously  monitor  progress  toward  sufficient  coverage  of  reference  model  practices  in 
preparation  for  rating.  Estimates  of  the  additional  data  collection  effort  should  be  reviewed 
regularly.  If  the  feasibility  of  the  appraisal  schedule  is  called  into  question,  a  replanning  effort 
might  be  necessary  (as  described  in  activity  2.1.5.3  Replan  Data  Collection). 

2.2.4  Generate  Appraisal  Results 

Purpose — Rate  goal  satisfaction  based  on  the  extent  of  practice  implementation  throughout 
the  appraised  organization.  The  extent  of  practice  implementation  is  judged  based  on 


CMU/SEI-2005-SR-001 


89 


validated  data  (e.g.,  direct,  indirect,  and  affirmation  objective  evidence)  collected  from  the 
entire  representative  sample  of  the  appraised  organization.  The  rating  of  maturity  levels  is 
driven  by  the  goal  satisfaction  ratings. 

Entry  Criteria — The  set  of  validated  preliminary  findings,  statements  of  practice 
implementation  gaps,  and/or  tabulations  of  validated  objective  evidence  of  practice 
implementation  on  which  they  are  based  are  available.  Team  members  are  confident  that  they 
have  obtained  all  the  data  they  will  need  to  make  rating  judgments.  The  data  obtained 
completely  covers  the  practices  within  the  defined  People  CMM  model  scope  and  the  entire 
representative  sample  selected  for  the  appraised  organization. 

Inputs  -Appraisal  data,  including  the  following: 

•  validated  preliminary  findings 

•  tabulations  of  objective  evidence  of  practice  implementation 

•  annotated  worksheets,  checklists,  and  working  notes 

Activities — The  four  activities  listed  below  are  required  for  this  process. 

2.2.4. 1  Derive  Findings  and  Rate  Goals 

2.2.4.2  Determine  Satisfaction  of  Process  Areas 

2.2.4.3  Determine  Maturity  Level 

2.2.4.4  Document  Appraisal  Results 

Outputs 

•  final  findings 

•  recorded  rating  decisions 

Outcome — A  formal  rating  decision  is  needed  for  each  reference  model  component  to  be 
rated,  and  for  which  the  team  obtained  complete  or  sufficient  data. 

Exit  Criteria— Ratings  against  all  components  per  the  plan  have  been  made  and  recorded. 

Key  Points — The  judgment  of  goal  satisfaction  is  based  on  and  traceable  to  the  extent  of  the 
implementation  of  practices  associated  with  that  goal  (or  alternative  practices  contributing 
equivalently  to  goal  satisfaction). 

Success  in  this  activity  is  driven  by  team  members’  ability  to  limit  their  focus  to  the  data  that 
support  the  judgments  and  to  avoid  issues  that  threaten  their  ability  to  be  objective.  This 
activity  can  create  a  great  deal  of  stress  for  team  members  under  pressure  to  help  their 
organization  do  well.  The  team  leader  must  skillfully  facilitate  this  activity  when  external 
•pressures  exist. 

Tools  and  Techniques— There  is  a  significant  amount  of  data  to  review  in  making  each 
round  of  judgments.  Rating  worksheets  and  automated  support  tools  facilitate  the  team’s 
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decision-making  process  by  presenting  necessary  data  in  a  concise,  well-organized  manner. 
When  controversial  issues  are  encountered,  the  team  leader  must  actively  ensure  that  the 
team  remains  focused  on  the  pertinent  issues.  Strategic  rest  breaks  and  sequencing  and  pacing 
critical  discussions  are  often  keys  to  success. 

Metrics 

•  planned  versus  actual  effort  for  each  component  rated 

•  number  of  model  components  rated  satisfied  or  unsatisfied 

Metrics  are  captured  during  the  appraisal  and  documented  in  the  appraisal  report. 

Verification  and  Validation — The  team  leader  verifies  that  the  rating  process  was  performed 
in  accordance  with  the  method  rules  and  the  rating  baseline  selected  and  documented  in  the 
appraisal  plan.  Work  aids  used  to  record  the  team  judgments  help  ensure  traceability  to  the 
basis  for  the  rating  judgments. 

Records — A  worksheet  or  other  work  aid  might  be  used  to  make  a  record  of  the  rating 
decisions. 

Tailoring — The  method  provides  tailoring  options  for  rating  additional  model  components. 
The  minimum  requirement  is  to  rate  the  goals  associated  with  each  PA  in  the  scope  of  the 
appraisal.  In  addition,  the  sponsor  might  request  that  maturity  level  ratings  be  performed  and 
reported.  Through  negotiation  between  the  appraisal  team  leader  and  the  appraisal  sponsor,  a 
decision  to  rate  individual  practices  can  also  be  made  when  requested  by  the  sponsor  or 
accepted  by  the  appraisal  team  as  adding  value  to  the  final  findings. 

The  characterization  of  practices  (see  activity  22.2.2  Characterize  Implementation  of  Model 
Practices)  leads  to  a  rating  of  practices,  when  done  following  method  requirements. 

Interfaces  with  Other  Processes — The  rating  judgments  made  by  the  appraisal  team 
members  are  dependent  on  the  quality  of  the  data  available  to  them  and  their  ability  to 
reliably  judge  the  implementation  and  institutionalization  of  practices  in  the  organization  that 
relate  to  the  People  CMM  model.  All  the  processes  previously  described  contribute  to  the 
team’s  ability  to  effectively  execute  this  process. 

Process  2.1.1  Analyze  Requirements  establishes  the  rating  baseline,  the  appraised 
organization  to  which  ratings  will  apply,  and  the  purpose  for  which  the  ratings  will  be  used. 
Process  2.1.2  Develop  Appraisal  Plan,  in  conjunction  with  the  process  2.1.4  Obtain  and 
Analyze  Initial  Objective  Evidence  and  process  2.1.5  Prepare  for  Collection  of  Objective 
Evidence,  determines  the  sample  of  the  appraised  organization  for  which  data  will  be 
collected  and  the  ratings  will  be  determined. 

Process  2.1.3  Select  and  Prepare  Team  ensures  that  the  team  has  sufficient  knowledge  and 
skills  to  interpret  the  data  and  arrive  at  sound  rating  judgments.  Processes  2.2. 1  Examine 
Objective  Evidence  and  2.2.3  Document  Objective  Evidence  provide  the  basic  information 
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needed  to  support  making  the  judgments.  Process  2.2.2  Verify  and  Validate  Objective 
Evidence  characterizes  the  extent  to  which  the  organization  implements  practices  in  the 
model  (or  acceptable  alternatives)  and  validates  findings  describing  associated  weaknesses. 
Upon  the  successful  execution  of  these  processes,  the  team  is  ready  to  rate  the  satisfaction  of 
goals  dependent  on  those  practices. 

Summary  of  Activities — The  required  rating  activity  involves  making  team  judgments  about 
goal  satisfaction  for  each  and  every  goal  within  the  model  scope  defined  in  the  rating 
baseline.  Once  goal  satisfaction  has  been  determined,  optional  rating  activities  can  be 
performed  in  accordance  with  the  defined  rating  baseline  and  the  model.  The  first  optional 
activity  focuses  on  rolling  up  goal  satisfaction  to  PA  ratings.  The  team  assigns  a  maturity 
level  rating  (1  through  5)  corresponding  to  the  highest  level  in  the  model  for  which  all 
applicable  PAs  have  been  rated  as  satisfied. 

2.2.4.1  Derive  Findings  and  Rate  Goals 

Activity  Description — The  judgments  made  about  goal  satisfaction  are  driven  by  the 
findings  that  were  documented  by  the  appraisal  team  and  validated  by  appraisal  participants. 
The  preliminary  findings  focus  on  gaps  in  the  implementation  of  practices.  When  performing 
goal  ratings,  the  team  must  judge  whether  or  not  these  gaps  in  the  implementation  of 
practices  (in  aggregate)  threaten  the  organization’s  ability  to  satisfy  the  goals  associated  with 
the  practices. 

Required  Practices 

•  Derive  final  findings  using  preliminary  findings  statements,  feedback  from  validation 
activities,  and  any  additional  objective  evidence  collected  as  a  result  of  the  validation 
activity. 

•  Rate  each  goal  within  the  reference  model  scope  of  the  appraisal  based  on  the  practice 
implementation  characterizations  at  the  appraised  organization  level  and  the  aggregation 
of  weaknesses  associated  with  that  goal. 

•  Obtain  appraisal  team  consensus  on  the  practice  implementation  characterizations, 
findings  statements,  and  ratings  generated  for  the  appraised  organization  level. 

Parameters  and  Limits — When  deriving  final  findings,  the  aim  is  to  create  goal-level 
statements  that  summarize  the  gaps  in  practice  implementation.  These  statements  must  be 
abstracted  to  the  level  of  the  appraised  organization  and  cannot  focus  on  individual  units, 
unless  the  tailoring  option  for  unit-specific  findings  has  been  agreed  upon  during  planning. 

If  there  are  no  findings  that  document  the  weaknesses  associated  with  a  goal,  the  goal  must 
be  satisfied.  The  goal  is  rated  satisfied  if  both  of  the  following  occur: 

•  All  associated  practices  are  characterized  at  the  appraised  organization  level  as  either 
largely  implemented  or  fully  implemented. 
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•  The  aggregation  of  weaknesses  associated  with  the  goal  does  not  have  a  significant 
negative  impact  on  goal  achievement. 

For  a  goal  to  be  rated  as  unsatisfied,  the  team  must  be  able  to  describe  how  the  set  of 
weaknesses  (or  single  weakness)  led  to  this  rating. 

Optional  Practices — Findings  statements  and  satisfaction  ratings  may  be  specified  at  the 
level  of  individual  practices  if  the  appraisal  sponsor  specifically  requests  this  tailoring  option. 
These  practice-level  ratings  must  be  based  on  the  extent  to  which  the  implemented  practice 
(or  the  absence  of  implementation)  supports  the  achievement  of  the  related  goal.  The  use  of 
informative  material  to  form  a  checklist  is  explicitly  discouraged  since  it  might  cloud 
discovery  of  valid  information  in  the  organization.  A  rating  algorithm  for  practices  that  does 
not  have  a  demonstrable  link  to  PA  goals  would  depart  from  the  intended  use  of  People  CMM 
components. 

Implementation  Guidance — Any  endeavor  that  results  in  a  score,  grade,  or  rating  is  by 
definition  an  area  of  sensitivity  to  those  affected  by  the  outcome.  An  objective  and  clear-cut 
basis  for  assigning  a  rating  lessens  this  sensitivity  and  results  in  a  more  consistent  basis  of 
comparison  among  the  appraised  organizations  and  goals  rated.  Judgments  made  prior  to  and 
during  the  rating  process  should  be  based  on  observable  facts  and  should  be  made  at  the 
lowest  level  of  abstraction  that  makes  sense.  In  the  case  of  People  CMM,  the  lowest  level  of 
abstraction  is  characterizing  the  extent  of  practice  implementation  for  each  process 
instantiation  within  the  representative  sample.  Characterizations  made  at  the  instantiation 
level  are  aggregated  into  a  characterization  of  the  extent  of  practice  implementation 
throughout  the  organization,  as  described  earlier  in  process  2.2.2  Verify  and  Validate 
Objective  Evidence.  The  judgment  of  goal  satisfaction  is  then  based  on  and  directly  traceable 
to  the  extent  of  implementation  of  practices  associated  with  that  goal  (or  alternative  practices 
contributing  equivalently  to  goal  satisfaction). 

Findings  should  be  phrased  in  terms  that  help  the  appraisal  sponsor  make  decisions  and  take 
action  based  on  the  appraisal  results. 

Final  findings  are  best  evolved  from  the  evidence  and  statements  of  strengths  or  weaknesses 
captured  in  the  process  area  workbook  tracking  tool.  Where  possible,  characterize  each 
practice  to  make  the  goal  rating  more  evident. 

2.2A.2  Determine  Satisfaction  of  Process  Areas 

Activity  Description — When  using  the  People  CMM  model,  the  team  may  derive  the 
satisfaction  of  PAs  from  the  set  of  goal  satisfaction  judgments.  Assigning  PA  satisfaction 
ratings  is  an  optional  activity,  selected  at  the  discretion  of  the  appraisal  sponsor  and 
documented  in  the  appraisal  input. 

Required  Practices — Rate  the  satisfaction  of  each  PA  in  the  scope  of  the  appraisal  based  on 
the  ratings  of  the  goals  within  each  PA,  if  this  rating  option  was  selected  during  planning. 


CMU/SEI-2005-SR-001 


93 


All  ratings  are  captured  in  the  process  area  workbook  tracking  tool. 

Parameters  and  Limits — PAs  might  be  assigned  rating  values  of  satisfied,  unsatisfied,  not 
applicable,  or  not  rated. 

A  PA  is  rated  satisfied  if,  and  only  if,  all  of  its  goals  are  rated  satisfied.  If  even  one  of  the 
goals  in  a  PA  is  rated  unsatisfied,  then  the  PA  is  rated  unsatisfied.  When  a  PA  is  determined  to 
be  outside  of  the  appraised  organization’s  scope  of  work,  the  PA  is  designated  not  applicable 
and  is  not  rated.  The  identification  of  a  PA  as  not  applicable  must  occur  during  the  planning 
of  the  appraisal. 

When  a  PA  is  outside  of  the  appraisal  scope,  or  if  the  associated  set  of  objective  evidence 
does  not  meet  the  defined  criteria  for  sufficient  data  coverage,  the  PA  is  not  rated  and 
designated  as  such.  The  criteria  for  sufficient  data  coverage  are  described  in  activity  2.2.2. 1 
Verify  Objective  Evidence. 

Optional  Practices — A  profile  to  summarize  the  satisfaction  of  goals  might  be  created  to 
provide  further  insight  about  the  rating  outcomes.  Where  a  PA  is  rated  as  unsatisfied,  this 
more  detailed  view  of  the  rating  outcomes  might  provide  focus  and  visibility  at  a  lower  level 
of  detail. 

Implementation  Guidance— PA  satisfaction  is  a  direct  function  of  goal  satisfaction.  A  PA  is 
rated  as  satisfied  if  every  goal  contained  in  the  PA  is  rated  as  satisfied.  A  PA  is  rated  as 
unsatisfied  if  any  goal  is  rated  as  unsatisfied.  This  ensures  that  one  or  more  weaknesses  exist 
that  serve  to  explain  why  the  goal,  and  therefore  the  PA,  is  not  satisfied. 

PA  ratings  need  not  be  reported  to  appraisal  participants  if  the  sponsor  does  not  wish  to 
disclose  them.  However,  a  documented  output  from  this  rating  activity,  if  it  is  performed,  is  a 
required  component  in  the  appraisal  record. 

2.2A.3  Determine  Maturity  Level 

Activity  Description — Historically,  one  of  the  most  visible  outcomes  of  an  appraisal  has 
been  the  maturity  level  rating  assigned.  The  determination  of  a  maturity  level  rating  is 
straightforward  and  derived  mechanically  from  the  ratings  assigned  at  the  lower  levels. 
Assigning  a  maturity  level  rating  for  Class  A  appraisals  is  an  optional  activity  selected  at  the 
discretion  of  the  appraisal  sponsor  and  documented  in  the  appraisal  input. 

Required  Practices — Rate  the  maturity  level  based  on  the  ratings  assigned  to  PAs,  if  this 
rating  option  was  selected  during  planning. 

All  ratings  are  captured  in  the  process  area  workbook  tracking  tool. 

Parameters  and  Limits— The  maturity  level  determined  is  the  highest  level  at  which  all  PAs 
contained  within  the  maturity  level  and  all  lower  maturity  levels  are  rated  as  satisfied  or  not 
applicable. 
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Optional  Practices — None. 

Implementation  Guidance — This  activity  may  be  omitted  since  it  is  a  tailoring  option.  If  a 
maturity  level  is  to  be  reported,  the  PA  ratings  that  form  the  basis  for  the  maturity  level  rating 
are  derived  as  described  in  activity  2.2A.2  Determine  Satisfaction  of  Process  Areas. 

2.2.4.4  Document  Appraisal  Results 

Activity  Description — The  results  of  the  appraisal  conduct  must  be  documented  for 
reporting.  Verbal  reports  of  the  rating  outcomes  or  face-to-face  explanations  of 
implementation  gaps  discovered  by  the  team  are  not  sufficient  to  communicate  appraisal 
results. 

Required  Practices 

•  Document  the  final  findings. 

•  Document  the  rating  outcome(s). 

•  Document  the  appraisal  disclosure  statement  (ADS). 

Parameters  and  Limits- — The  ADS  and  the  set  of  appraisal  outputs  agreed  upon  with  the 
appraisal  sponsor  must  be  documented.  These  appraisal  outputs  might  exclude  all  ratings,  and 
the  sponsor  is  free  to  select  and  disclose  a  variety  of  appraisal  outcomes  (as  specified  in  the 
activities  of  this  process). 

Regardless  of  the  needs  of  the  sponsor,  the  ADS,  goal  ratings,  and  associated  findings  must 
be  documented  as  a  part  of  the  appraisal  information  returned  to  the  People  CMM  Steward. 

Optional  Practices — Any  optional  outputs  requested  by  the  appraisal  sponsor  are  also 
created  during  this  activity. 

Implementation  Guidance — This  activity  is  focused  on  collecting  and  documenting  the 
results  of  prior  activities  related  to  the  generation  of  findings  and  ratings.  Depending  on  the 
planned  recipients  of  the  results,  multiple  forms  of  the  results  might  be  needed.  Certain  data 
might  not  be  appropriate  for  all  audiences,  or  the  style  or  language  of  the  results  might  need 
to  be  adjusted  to  best  fit  the  needs  of  the  recipients. 

The  documented  appraisal  results  are  typically  provided  in  a  final  findings  presentation, 
described  in  activity  2.3. 1.1  Present  Final  Findings. 

The  final  findings  can  be  developed  directly  from  the  process  area  workbook  tracking  tool. 


2.3  Phase  3 

This  phase  has  two  processes:  Deliver  Appraisal  Results,  and  Package  and  Archive  Appraisal 
Assets. 
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2.3.1  Deliver  Appraisal  Results 

Purpose— Provide  credible  appraisal  results  that  can  be  used  to  guide  actions.  Represent  the 
strengths  and  weaknesses  of  the  processes  in  use  at  the  time.  Provide  ratings  (if  planned  for) 
that  accurately  reflect  the  maturity  level  of  the  processes  in  use. 

Entry  Criteria 

•  Objective  evidence  has  been  validated  through  the  team  process. 

•  Preliminary  findings  have  been  validated. 

•  Ratings  have  been  determined  (for  model  components  selected  for  rating). 

•  Final  findings  have  been  created  and  reviewed  by  the  team. 

Inputs 

•  appraisal  data 

-  final  findings 

-  ratings 

•  appraisal  artifacts 

-  appraisal  input 

-  appraisal  plan 

Activities — The  three  activities  listed  below  are  required  for  this  process. 

2.3. 1.1  Present  Final  Findings 

2.3. 1.2  Conduct  Executive  Session(s) 

2.3. 1.3  Plan  for  Next  Steps 

Outputs 

•  documented  final  findings 

•  final  report  (if  requested) 

•  recommendations  report  (if  requested) 

Outcome 

•  The  results  of  the  appraisal  are  provided  to  the  sponsor  and  the  appraised  organization. 

•  A  valid  and  reliable  characterization  of  the  current  state  of  the  processes  in  use  across  the 
appraised  organization  is  documented. 

Exit  Criteria 

•  Appraisal  results  are  delivered  to  the  appraisal  sponsor  and  appraised  organization. 

•  •  An  executive  session  is  conducted,  if  appropriate. 

Key  Points — The  appraisal  results  are  intended  to  support  decision  making  and  need  to  be 
delivered  in  a  way  that  promotes  appropriate  actions.  Whether  the  appraisal  was  conducted 
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for  internal  process  improvement  or  process  monitoring  purposes,  the  delivery  of  results 
should  facilitate  the  actions  that  are  driven  by  the  information. 

Tools  and  Techniques — Templates  containing  standard  information  for  use  in  a  final 
findings  briefing  are  provided  to  all  SCAMPI  with  People  CMM  Lead  Appraisers. 
Experienced  appraisers  frequently  use  electronic  (database)  tools  that  support  the 
transformation  of  raw  appraisal  data  into  appraisal  results.  These  tools  might  be  useful  in  real 
time  as  appraisal  results  are  presented.  Strategies  for  presenting  and  packaging  the  results 
should  leverage  presentation  and  documentation  techniques  that  best  suit  the  audience. 

Metrics— It  is  highly  recommended  that  the  attendance  at  the  final  briefing  (if  one  is  held)  be 
recorded.  Significant  absenteeism  of  key  stakeholders  is  likely  to  be  an  indication  of  risk  for 
future  success  in  addressing  the  appraisal  findings. 

Metrics  are  captured  at  the  meeting  and  documented  in  the  ADS. 

Verification  and  Validation — The  required  elements  of  appraisal  results  are  specified  in  the 
activity  description,  and  a  checklist  helps  verily  that  these  elements  are  present.  Validation  of 
this  activity  can  only  occur  after  the  appraisal  is  complete. 

Records 

•  final  findings 

•  final  report  (if  requested) 

•  recommendations  report,  if  required  by  sponsor 

Tailoring — In  some  internal  process  improvement  usage  of  the  method,  the  executive  session 
might  be  tailored  out.  The  appraisal  sponsor  should  make  this  decision  with  the  full 
involvement  of  the  appraisal  team  leader. 

Interfaces  with  Other  Processes— Upon  completion  of  process  2.2.4  Generate  Appraisal 
Results,  the  ratings  and  findings  generated  are  used  to  prepare  and  deliver  the  final  appraisal 
results  to  the  appraisal  sponsor  and  appraised  organization.  The  appraisal  results  become  part 
of  the  appraisal  record,  which  is  discussed  in  process  2.3.2  Package  and  Archive  Appraisal 
Assets. 

Summary  of  Activities — The  final  findings  contain  the  validated  strengths,  weaknesses,  and 
ratings  (as  defined  by  the  appraisal  plan)  reflecting  the  organizational  process  maturity  level 
for  PAs  within  the  appraisal  scope.  Other  appraisal  outputs,  as  requested  by  the  appraisal 
sponsor  and  documented  in  the  appraisal  plan,  are  generated  and  provided.  Optionally,  a 
separate  executive  session  might  also  be  held  to  clarify  and  discuss  the  appraisal  results  from 
a  senior  management  perspective  to  facilitate  decision  making.  Plans  are  established  for 
acting  on  the  appraisal  results. 
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2.3.1. 1  Present  Final  Findings 

Activity  Description — The  final  findings  contain  a  summary  of  the  strengths  and 
weaknesses  for  each  PA  within  the  appraisal  scope,  as  well  as  additional  information  that 
provides  context  for  the  findings.  The  generation  of  the  findings  is  addressed  in  activity 

2.2.4. 1  Derive  Findings  and  Rate  Goals.  The  presentation  to  the  appraisal  sponsor  and 
appraised  organization  might  be  in  a  summarized  form  with  the  detailed  findings  provided  as 
backup  information.  Final  findings  are  often  presented  using  view  graphs  in  a  meeting  room 
or  auditorium. 

In  addition  to  the  final  findings,  a  draft  ADS  summarizing  the  results  of  the  appraisal  is 
provided  to  the  appraisal  sponsor. 

Required  Practices 

•  Provide  appraisal  final  findings  to  the  appraisal  sponsor  and  the  appraised  organization. 

•  Provide  an  ADS  to  the  appraisal  sponsor  summarizing  the  appraisal  results  and 
conditions  under  which  the  appraisal  was  performed. 

Parameters  and  Limits— Required  elements  of  the  final  findings  include  the  following: 

•  summary  of  the  appraisal  process 

•  findings  (summary  of  strengths  and  weaknesses) 

Appraisal  team  consensus  must  be  obtained  on  the  wording  of  the  final  findings  to  ensure 
that  the  whole  team  supports  the  accuracy  of  the  described  appraisal  results. 

The  team,  when  delivering  the  final  findings,  must  adhere  to  some  important  principles,  listed 
below. 

•  If  a  model  component  is  reported  as  unsatisfied,  the  corresponding  findings  of 
weaknesses  that  caused  the  team  to  make  that  judgment  must  also  be  reported. 

•  Confidentiality  and  non-attribution  principles  apply  to  statements  made  in  the 
presentation  of  final  findings. 

The  ADS  is  a  summary  statement  describing  the  appraisal  results  that  includes  the  conditions 
and  constraints  under  which  the  appraisal  was  performed.  It  contains  information  considered 
essential  to  adequately  interpret  the  meaning  of  assigned  maturity  level  ratings.  The  ADS  is 
prepared  by  the  appraisal  team  leader  and  provided  to  the  appraisal  sponsor.  Otherwise,  the 
appraisal  team  leader  delivers  the  ADS  to  the  sponsor  as  a  separate  document. 

A  detailed  description  of  the  ADS  contents  is  provided  in  Appendix  A.  The  ADS  is 
considered  a  draft  at  this  stage  of  the  appraisal  process.  The  ADS  must  also  contain  an 
•  affirmation  that  all  appraisal  requirements  have  been  satisfied,  which  cannot  be  claimed  until 
all  appraisal  activities  are  complete. 

Optional  Practices — Optional  elements  of  the  final  findings  report  include  the  following: 


•  ratings 

•  improvement  activities 

•  recommended  actions,  if  requested  by  the  sponsor 

•  schedule  of  major  upcoming  events  (e.g.,  appraisal  report,  recommendations,  action  plan, 
reappraisal),  if  required 

Note  that  the  generation  of  goal  ratings  by  the  appraisal  team  is  required  (as  described  in 
process  2.2.4  Generate  Appraisal  Results).  However,  these  ratings  may  be  excluded  from  the 
final  findings  at  the  discretion  of  the  appraisal  sponsor. 

A  formal  presentation  of  appraisal  results  by  the  appraisal  team  is  frequently  the  final  visible 
activity  for  appraisals  conducted  for  internal  process  improvement.  The  final  findings 
presentation  typically  is  delivered  in  the  form  of  a  face-to-face  briefing  at  the  end  of  the 
appraisal  on-site  period. 

Other  mechanisms  for  providing  the  appraisal  results  to  the  appraised  organization,  such  as 
written  reports,  might  be  more  practical  when  using  the  method  for  process  monitoring. 
Although  the  timeframe  for  providing  appraisal  results  might  vary,  the  appraisal  cannot  be 
considered  complete  until  the  final  findings  are  provided. 

The  draft  ADS  may  be  provided  during  the  executive  sessions  (if  performed),  at  the 
conclusion  of  the  final  findings  briefing,  or  at  some  other  agreed  upon  date. 

Implementation  Guidance — The  People  CMM  Steward  provides  Lead  Appraisers  with  a 
template  for  a  final  findings  briefing  describing  typical  contents  and  format. 

Findings  include  a  summary  of  strengths  and  weaknesses  determined  for  each  PA  within  the 
appraisal  reference  model  scope.  This  might  also  include  global  findings  that  apply  across 
multiple  PAs  and  non-reference  model  findings  that  affect  the  implementation  (positively  or 
negatively)  of  associated  processes  within  the  appraised  organization. 

Normally,  the  appraisal  team  leader  presents  the  final  findings.  In  some  applications  of  the 
method  for  internal  process  improvement,  the  team  might  elect  to  have  an  appraisal  team 
member  or  members  from  the  appraised  organization  provide  the  briefing  to  encourage  the 
acceptance  of  the  final  findings  and  ownership  of  the  appraisal  results  for  follow-on  action. 
The  latter  is  a  preferred  approach  if  the  ATL  does  not  work  in  the  appraised  organization  but 
the  appraisal  team  members  do. 

As  a  courtesy,  the  appraisal  team  can  inform  the  appraisal  sponsor  and/or  the  senior  site 
manager  of  the  appraisal  results  prior  to  presenting  them  publicly  in  the  final  findings 
briefing.  This  might  help  them  to  avoid  surprises  and  obtain  feedback  on  ways  to  present  the 
findings  that  best  meet  the  needs  of  the  sponsor,  appraisal  participants,  and  the  appraised 
organization.  See  activity  2.3. 1.2  Conduct  Executive  Session(s)  for  a  description  of  topics  for 
discussion. 
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The  number  and  scope  of  findings  reported  will  affect  the  impact  of  appraisal  results,  whether 
or  not  the  team  intends  for  this  to  happen.  There  are  times  when  providing  a  long  list  of 
details  is  beneficial.  Other  times,  high-level  summaries  are  more  appropriate. 


2.3.1 .2  Conduct  Executive  Session(s) 

Activity  Description— The  executive  session  is  an  optional  activity  that  may  be  performed  at 
the  discretion  of  the  appraisal  sponsor  or  senior  site  manager.  The  executive  session  provides 
the  appraisal  sponsor,  senior  site  manager,  and  invited  staff  with  a  private  opportunity  to  (a) 
discuss  with  the  appraisal  team  leader  any  issues  with  the  appraisal,  (b)  obtain  clarification  of 
the  appraisal  results,  (c)  confirm  understanding  of  the  process  issues,  and  (d)  provide 
guidance  about  the  focus,  timing,  and  priorities  of  the  recommendations  report  and  follow-on 
activities. 

Required  Practices— None.  If  the  option  is  selected,  hold  a  private  meeting  between  the 
appraisal  team  leader  and  the  sponsor,  including  any  participants  invited  by  the  sponsor. 

The  executive  session  does  not  have  to  be  held  if  the  sponsor  does  not  require  it.  The 
appraisal  sponsor  should  make  this  decision  with  the  full  involvement  of  the  appraisal  team 
leader. 

Parameters  and  Limits — If  an  executive  session  is  conducted,  the  confidentiality  and  non¬ 
attribution  of  data  sources  must  be  maintained.  Multiple  sessions  may  be  held  if  necessary, 
targeted  at  the  information  needs  of  the  executive  audience. 

Optional  Practices — Attendance  of  the  entire  appraisal  team  at  the  executive  sessions  is  a 
tailoring  option.  The  executive  session  is  also  an  appropriate  opportunity  to  review  appraisal 
performance  with  the  appraisal  sponsor  and/or  senior  site  manager,  including  planned  versus 
actual  execution  of  the  appraisal  plan,  including  method  tailoring.  This  provides  additional 
input  on  the  appropriate  expectations  for  interpreting  and  handling  the  appraisal  results. 

The  draft  ADS  can  be  provided  during  the  executive  session  instead  of  at  the  conclusion  of 
the  final  findings  briefing  as  discussed  in  activity  2.3. 1.1  Present  Final  Findings. 

Implementation  Guidance — The  executive  sessions  ensure  that  the  appraisal  sponsor  and/or 
the  senior  site  manager  have  a  sound  understanding  of  the  appraisal  results.  Any  feedback 
obtained  from  these  executive  sessions  should  be  recorded.  All  confidentiality  and  non¬ 
attribution  rules  are  still  in  effect. 

2.3.1 .3  Plan  for  Next  Steps 

Activity  Description— After  the  delivery  of  the  appraisal  results,  a  plan  for  follow-on 
activities  is  determined.  The  planned  follow-on  activities  are  typically  defined  in  the 
appraisal  plan,  reflecting  sponsor  requests  for  additional  appraisal  tasks  and  products 
necessary  to  meet  appraisal  objectives  or  a  commitment  to  take  action  upon  the  appraisal 
results.  Follow-on  activities  might  include  the  following: 
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•  development  of  a  final  report 

•  development  of  a  recommendations  report  or  briefing 

•  generation  or  update  of  a  process  improvement  plan 
Required  Practices — None. 

Parameters  and  Limits — None. 

Optional  Practices — Planning  for  next  steps  is  an  optional,  but  recommended,  appraisal 
activity. 

Implementation  Guidance — Planning  for  next  steps  includes  optional  activities  such  as  the 
following: 

•  development  of  a  final  report  by  the  appraisal  team  summarizing  the  appraisal  results  for 
the  appraisal  sponsor 

•  submission  of  appraisal  team  recommendations  for  action  upon  the  appraisal  findings 

•  generation  of  a  process  improvement  action  plan  for  the  appraised  organization 

In  addition  to  specifying  the  activities  to  be  performed,  these  plans  usually  include  the 
assignment  of  responsibility,  schedule,  and  estimated  resources  for  the  implementation  of  the 
follow-on  actions.  The  plans  established  can  be  used  to  track  the  progress  of  the  follow-on 
activities  over  time. 

Implementation  Guidance:  Process  Improvement  Action  Planning — Findings  and 
recommendations  from  the  appraisal  team  can  be  used  by  the  appraised  organization  to 
establish  action  plans  for  process  improvement.  This  is  an  optional  output  most  often  used  in 
internal  process  improvement  or  process-monitoring  applications  of  the  appraisal  method. 

Recommendations  often  include  a  prioritized  list  of  improvement  activities,  such  as  an 
improvement  plan  that  defines  the  tasks,  schedules,  and  resources  necessary  for 
implementation. 

Follow-on  appraisals  are  usually  performed  to  verify  improvement  progress.  This  could 
include  a  combination  of  Class  A,  B,  and  C  appraisals  (refer  to  the  ARC  for  additional  details 
[SEIOlb]). 

Implementation  Guidance:  Final  Report — The  final  report  provides  details  or  explanations 
beyond  what  was  contained  in  the  final  findings.  The  generation  of  an  appraisal  final  report  is 
an  optional  activity  that,  if  requested  by  the  appraisal  sponsor,  documents  the  execution  of  the 
appraisal,  contains  detailed  appraisal  findings,  and  forms  a  basis  for  action  planning.  This 
baseline  is  used  for  subsequent  reports  and  follow-on  actions  and  might  also  be  an  input  in 
subsequent  appraisals. 

Items  contained  or  referenced  in  the  final  report,  either  in  their  entirety  or  as  a  subset,  might 
include  the  following: 
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•  executive  summary  of  the  appraisal  process  and  results 

•  appraisal  input  (see  process  2.1.1  Analyze  Requirements) 

•  appraisal  plan  (see  process  2.1.2  Develop  Appraisal  Plan) 

•  appraisal  record  (see  process  2.3.2  Package  and  Archive  Appraisal  Assets) 

The  final  report  should  be  completed  as  soon  as  possible  after  the  appraisal,  preferably  within 
four  weeks.  The  appraisal  team  leader  usually  generates  the  final  report  and  the  other  team 
members  contribute.  Usually,  team  members  are  assigned  sections  based  on  their  appraisal 
roles  or  mini-teams,  but  the  LA  is  responsible  for  managing,  not  generating,  the  report. 
Alternately,  the  team  could  write  the  report  and  the  Lead  Appraiser  review  it. 

The  format  and  content  of  the  final  report  varies  according  to  its  intended  use  by  the  appraisal 
sponsor.  In  its  simplest  form,  it  could  be  a  set  of  notes  annotated  to  the  final  findings, 
elaborating  on  some  aspect  of  the  findings  or  capturing  essential  comments  or 
recommendations  from  the  appraisal  team.  Or,  the  final  report  could  be  completed  with  the 
final  findings. 

Implementation  Guidance:  Recommendations  Report — If  requested  by  the  appraisal 
sponsor,  appraisal  team  recommendations  for  taking  action  on  the  appraisal  results  can  be 
provided.  These  recommendations  can  provide  momentum  to  the  appraisal  follow-up  by 
serving  as  a  link  between  the  appraisal  findings  and  subsequent  decision  making  or  action 
plans.  The  emphasis  of  these  recommendations  depends  on  the  appraisal  sponsor’s  objectives 
and  planned  use  of  the  appraisal  results,  as  defined  in  the  appraisal  input.  This  can  vary 
widely  based  on  the  context  in  which  the  appraisal  method  is  applied  (e.g.,  internal  process 
improvement,  process  monitoring). 

The  recommendations  report  should  be  completed  as  soon  as  possible  after  the  appraisal  on¬ 
site  period.  Depending  on  the  nature,  complexity,  and  use  of  the  recommendations,  it  might 
take  as  long  as  two  months  to  produce.  Rather  than  generate  a  separate  recommendations 
report,  a  common  alternative  practice  is  to  include  these  recommendations  in  the  final  report. 

Consider  the  possibility  that  the  expertise  needed  for  making  the  appropriate 
recommendations  might  be  beyond  the  level  of  expertise  on  the  appraisal  team. 

2.3.2  Package  and  Archive  Appraisal  Assets 

Purpose— Preserve  important  data  and  records  from  the  appraisal,  and  dispose  of  sensitive 
materials  in  an  appropriate  manner. 

Entry  Criteria 

'•  Appraisal  has  been  conducted. 

•  Results  have  been  delivered  to  the  sponsor. 

•  All  appropriate  data  have  been  collected  and  retained  during  the  appraisal. 
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Inputs 

•  appraisal  data 

-  appraisal  input 

-  appraisal  plan 

-  final  findings 

-  objective  evidence 

•  appraisal  team  artifacts 

-  notes 

-  documented  practice  implementation  gaps 

-  preliminary  findings 

-  document  library 

Activities — The  four  activities  listed  below  are  required  for  this  process. 

2.3.2. 1  Collect  Lessons  Learned 

2.3.2.2  Generate  Appraisal  Record 

2.3.2.3  Provide  Appraisal  Feedback  to  People  CMM  Steward 

2.3.2.4  Archive  or  Dispose  of  Key  Artifacts 

Outputs 

•  appraisal  record 

•  completed  forms  and  checklists 

•  sanitized  data  (as  appropriate  and  agreed  upon  during  planning) 

•  lessons  learned  for  the  appraisal  team  and  organization 

Outcome — Data  and  artifacts  are  appropriately  archived  or  destroyed.  The  team  has  captured 
lessons  and  data  to  help  improve  the  appraisal  process.  Requirements  for  providing  appraisal 
artifacts  to  stakeholders  and  the  People  CMM  Steward  are  met. 

Exit  Criteria 

•  Appraisal  assets  are  baselined  and  archived. 

•  Required  reports  are  delivered  to  the  appropriate  stakeholders. 

•  Artifacts  containing  sensitive  information  are  disposed  of  in  an  appropriate  manner. 

Key  Points — Protect  the  confidentiality  of  sensitive  data  while  distributing  and  archiving 
appraisal  assets.  Bundle  related  information  together  whenever  appropriate. 

Tools  and  Techniques — Electronic  (database)  tools  for  managing  appraisal  data  provide 
assistance  in  ensuring  the  integrity  of  baselines  and  repackaging  information  for  archival 
purposes.  Electronic  tools  allow  the  Lead  Appraiser  to  remove  traceability  information  so  that 
data  can  be  provided  to  the  appropriate  people  while  preserving  the  anonymity  of  the  data 
sources. 
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Electronic  tools  also  support  the  submission  of  appraisal  data  to  the  People  CMM  Steward. 
This  reduces  the  administrative  burden  and  facilitates  the  analysis  of  appraisal  method 
performance  data.  These  tools  also  provide  feedback  to  the  appraisal  community  about  the 
consolidated  analysis  results. 

Metrics — Although  archiving  and  reporting  the  metrics  associated  with  the  conduct  of  the 
appraisal  is  an  important  element  of  this  activity,  the  metrics  associated  with  the  conduct  of 
this  activity  itself  are  limited.  The  effort  and  calendar  time  consumed  are  collected  and 
compared  to  the  plan.  Some  appraisal  team  leaders  will  choose  to  maintain  personal  metrics 
associated  with  the  artifacts  described  in  this  activity. 

Verification  and  Validation — The  Lead  Appraiser  requirements  checklist  guides  the 
verification  of  the  list  of  artifacts  provided  to  the  People  CMM  Steward.  Validation  is 
provided  by  the  People  CMM  Steward  upon  receipt  of  the  appraisal  record. 

Records 

•  appraisal  record 

•  lessons  learned 

Tailoring — The  usage  mode  and  constraints  of  the  appraisal,  as  well  as  the  sensitivity  of  the 
data  and  planned  use  of  appraisal  results,  might  greatly  affect  the  degree  to  which  appraisal 
data  is  retained,  sanitized,  or  discarded. 

Interfaces  with  Other  Processes— The  final  process  in  the  appraisal,  this  process  is  about 
collecting,  packaging,  and  archiving  results  and  artifacts  produced  by  previous  processes  that 
must  become  part  of  the  appraisal  record.  Most  notably,  this  includes  the  appraisal  input, 
appraisal  plan,  and  appraisal  results. 

Additionally,  sensitive  or  proprietary  data  produced  by  other  appraisal  processes  must  be 
returned  to  the  appraised  organization  or  destroyed. 

Summary  of  Activities — This  process  includes  the  data  collection,  data  management,  and 
reporting  activities  necessary  to  close  out  the  appraisal.  Data  collected  throughout  the 
appraisal  is  consolidated  and  baselined  and  becomes  a  permanent  part  of  the  appraisal  record. 

2.3.2.1  Collect  Lessons  Learned 

Activity  Description — As  one  of  the  final  activities  in  wrapping  up  an  appraisal,  teams 
typically  record  lessons  learned  from  their  experience.  The  purpose  of  these  lessons  learned 
is  to  document  what  went  right,  what  went  wrong,  and  suggestions  or  recommendations  for 
improving  the  method  or  its  execution. 

The  collection  of  lessons  learned  is  a  recommended  activity  for  the  improvement  of  future 
appraisals  but  is  not  a  method  requirement. 

Required  Practices — None. 
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Parameters  and  Limits — Lessons  learned  must  adhere  to  the  same  principles  of 
confidentiality  and  non-attribution  as  other  appraisal  results. 

Optional  Practices — All  practices  related  to  the  collection  of  lessons  learned  are  optional 
but  recommended.  If  the  team  has  identified  potential  improvements  to  elements  of  the 
People  CMM  Product  Suite,  these  can  be  submitted  as  change  requests  to  the  People  CMM 
Steward. 

The  Lead  Assessor  should  provide  feedback  to  each  appraisal  team  member. 

Implementation  Guidance — Capturing  lessons  learned  is  often  done  as  a  group  at  the  end  of 
the  appraisal  while  the  appraisal  activities  are  fresh  in  team  members’  minds.  This  can  be 
supplemented  with  additional  input  from  team  members  upon  further  reflection,  if  necessary. 
Appraisal  team  leaders  forward  these  aggregate  lessons  learned  to  various  stakeholders  as 
appropriate,  but  always  to  the  other  team  members.  Team  leaders  and  members  often 
maintain  summary  lists  of  appraisal  best  practices  and  lessons  learned  as  a  mechanism  for 
continuous  learning  and  improvement,  and  these  are  used  as  a  . resource  for  planning 
subsequent  appraisals. 

2.3.2.2  Generate  Appraisal  Record 

Activity  Description — Appraisal  data  collected  throughout  the  appraisal  is  aggregated  and 
summarized  into  a  permanent  record  documenting  the  appraisal  conduct  and  results.  The 
appraisal  record  is  delivered  to  the  appraisal  sponsor  for  retention. 

Required  Practices 

•  Collect  and  baseline  the  appraisal  data  that  becomes  part  of  the  permanent  records 
provided  to  appraisal  stakeholders. 

•  Document  the  satisfaction  of  all  SCAMPI  with  People  CMM  requirements. 

•  Generate  the  appraisal  record  from  baselined  planning  and  execution  data  collected 
throughout  the  appraisal. 

•  Deliver  the  appraisal  record  to  the  appraisal  sponsor. 

Parameters  and  Limits — Required  contents  of  the  appraisal  record  include  the  following: 

•  dates  of  the  appraisal 

•  appraisal  input 

•  appraisal  plan 

•  objective  evidence,  or  identification  thereof,  sufficient  to  substantiate  goal-rating 
judgments 

•  characterizations  of  practice  implementation  determined  at  the  instantiation  level  and 
aggregated  at  the  appraised  organization  level 

•  identification  of  the  appraisal  method  (and  version)  used  along  with  any  tailoring  options 


CMU/SEI-2005-SR-001 


105 


•  final  findings 

•  all  ratings  rendered  during  the  appraisal  (goals,  PAs,  and  maturity) 

•  ADS 

Depending  on  the  recipient  and  intended  usage,  appraisal  data  might  be  subject  to  being 
sanitized  or  edited  in  order  to  comply  with  rules  for  non-attribution,  confidentiality, 
protection  of  proprietary  information,  and  applicable  laws,  regulations,  or  standards  (e.g., 
acquisition  regulations  or  security  classification).  Recipients  are  expected  to  place  the 
appropriate  limitations  on  the  access  and  use  of  the  provided  appraisal  data. 

The  appraisal  team  leader  documents  that  all  SCAMPI  with  People  CMM  requirements  were 
satisfied. 

Optional  Practices— The  appraisal  record  should  also  contain  any  additional  outputs 
requested  by  the  appraisal  sponsor,  as  agreed  to  during  appraisal  planning  and  documented  in 
the  appraisal  input  (e.g.,  reports  by  unit). 

Implementation  Guidance — The  actual  objective  evidence  (artifacts  or  portions  of  artifacts) 
need  not  be  part  of  the  appraisal  record,  but  an  identification  of  the  objective  evidence  is 
required.  This  might  be  implemented  by  providing  the  PIIs  that  were  used  as  the  basis  for 
characterizing  practice  implementation  decisions. 

Guidance  on  the  protection  of  appraisal  data  can  be  summarized  based  on  the  recipient  of  the 
data  as  follows: 

•  appraisal  sponsor:  replacement  of  specific  sources  (persons,  units)  with  non-attributable, 
general  identifiers  (e.g.,  numeric  codes  assigned  to  units,  roles,  or  data-gathering 
sessions) 

•  People  CMM  Steward:  same  as  for  appraisal  sponsor,  for  data  that  is  shared  by  both.  For 
data  that  is  provided  only  to  the  People  CMM  Steward,  the  data  collection  vehicles  (e.g., 
forms)  are  already  designed  to  observe  non-attribution  and  confidentiality  rules. 
Additionally,  supplied  data  might  be  subject  to  further  sanitization  to  comply  with 
acquisition  or  security-related  restrictions. 

•  senior  site  manager:  in  cases  where  the  appraised  organization  is  separate  from  the 
appraisal  sponsor,  the  appraised  organization  is  typically  provided  only  with  appraisal 
results  and  not  data  related  to  planning  and  decision  making  or  data  that  makes  use  of  the 
results. 

2.3.2.3  Provide  Appraisal  Feedback  to  People  CMM  Steward 

Activity  Description — Appraisal  data  required  by  the  People  CMM  Steward  is  collected  and 
'  reported.  This  includes  a  subset  of  the  contents  of  the  appraisal  record,  as  well  other  data 
used  by  the  Steward  to  aggregate  and  analyze  appraisal  performance  data  for  reporting  to  the 
community  and  monitoring  the  quality  of  performed  appraisals. 
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Required  Practices — Submit  the  completed  appraisal  report  as  required  by  the  People  CMM 
Steward. 

Parameters  and  Limits— The  People  CMM  Steward  defines  the  specific  set  of  data  required 
for  submission  at  the  completion  of  an  appraisal.  Submission  of  the  appraisal  report  is 
required  for  the  appraisal  to  be  recorded  in  the  steward’s  database  of  appraisal  results.  This  is 
also  a  requirement  established  by  the  steward  to  maintain  Lead  Appraiser  authorization. 

Optional  Practices — If  the  objective  evidence  is  available  in  electronic  form,  it  can  be 
included  as  part  of  the  appraisal  report  submitted  to  the  People  CMM  Steward. 

Implementation  Guidance — The  appraisal  team  leader  is  responsible  for  ensuring  that 
appraisal  feedback  required  by  the  People  CMM  Steward  is  collected  and  reported.  The 
People  CMM  Steward,  as  custodian  of  the  product  suite  and  the  SEI  Appraisal  Program,  has 
several  objectives  in  seeking  appraisal  feedback: 

•  characterization  of  the  state  of  the  practice  in  the  appraisal  community,  for  the  collection 
and  distribution  of  effective  appraisal  techniques 

•  analysis  of  reported  appraisal  data  to  obtain  an  understanding  of  appraisal  performance 
for  continuous  improvement 

•  quality  control  within  the  Appraisal  Program,  to  ensure  a  high  level  of  confidence  in  the 
accuracy  of  appraisal  results 

Summarized  results  determined  from  appraisal  data  collected  are  provided  periodically  to  the 
community.  The  format  and  mechanisms  for  the  submission  of  this  data  are  established  by  the 
People  CMM  Steward. 

2.3.2.4  Archive  or  Dispose  of  Key  Artifacts 

Activity  Description — After  the  various  reports  are  delivered  to  the  appropriate  stakeholders 
and  the  appraisal  assets  have  been  baselined,  the  team  leader  is  responsible  for  properly 
archiving  or  disposing  of  the  appraisal  data,  in  accordance  with  agreements  made  with  the 
sponsor  and  documented  in  the  appraisal  input.  The  team  librarian  (if  one  is  used )  ensures 
that  all  documentation  and  objective  evidence  provided  by  the  organization  is  returned  or  . 
disposed  of  properly.  Any  remaining  team  artifacts  or  notes  are  also  disposed  of  properly. 

Required  Practices 

•  Archive  or  dispose  of  key  artifacts  collected  by  the  appraisal  team. 

•  Return  objective  evidence  provided  by  the  appraised  organization. 

Parameters  and  Limits — Strict  non-attribution  policies  apply  in  all  usage  modes  of 
SCAMPI  with  People  CMM.  Confidentiality  and  non-disclosure  agreements  established  with 
the  appraisal  team  members  remain  in  effect. 

Optional  Practices — None. 
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Implementation  Guidance — How  the  records  are  preserved  or  disposed  of  depends  on  the 
usage  mode  of  the  method  and  the  appraisal  objectives.  Confidentiality  rules  might  differ  by 
application. 

Confidentiality  of  results  can  be  characterized  as  one  of  the  following: 

•  known  only  to  the  recipient  organization 

•  known  to  the  recipient  and  sponsor,  when  they  are  from  different  organizations 

•  known  to  anyone 

The  sponsor  is  solely  responsible  for  determining  the  confidentiality  with  which  the  appraisal 
results  are  maintained.  The  non-attribution  of  data  to  specific  individuals  is  the  responsibility 
of  the  appraisal  team.  The  recipient  organization  may  choose  to  make  the  results  known 
outside  the  organization  if  the  sponsor  agrees  and  this  decision  is  included  in  the  plan.  At  a 
high  level,  this  might  be  done  for  marketing  and  public  relations  reasons.  Disclosure  of 
results  should  include  the  context  and  constraints  under  which  the  appraisal  was  performed 
(e.g.,  reference  model  scope,  organizational  scope),  as  defined  by  the  ADS  described  in 
process  2.3.1  Deliver  Appraisal  Results. 

Any  annotations  about  objective  evidence  provided  to  the  organization  by  the  appraisal  team 
should  be  recorded  and  archived  for  use  in  process  improvement  actions  or  for  reuse  in 
subsequent  appraisals. 
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3  Conclusions 


In  this  pilot,  TCS  was  able  to  implement  a  common  appraisal  method  across  multiple  models 
and  across  a  large  worldwide  organization,  meeting  its  original  goals. 

The  interpretation  guide  presented  in  Section  2  was  used  during  the  four  mini-appraisal  pilots 
and  the  final  enterprise  Class  A  appraisal.  With  pilots,  premises  are  tested  and  the  boundaries 
of  applications  are  discovered.  As  a  result,  the  guide  was  evaluated  and  updated  throughout 
the  process,  resulting  in  a  stable  interpretation  guide  used  to  implement  a  Class  A  SCAMPI 
with  People  CMM  appraisal. 

3.1  Lessons  Learned  from  the  Mini-Appraisals 

Some  lessons  learned  during  the  four  mini-appraisal  pilots  include  the  following: 

1 .  The  SCAMPI  method  transfers  very  readily  to  the  People  CMM. 

2.  People  CMM  has  many  more  practices  than  CMMI  that  can  only  be  sufficiently  verified 
at  the  organization  level.  Thus  the  PIIs  developed  for  the  first  pilot  needed  to  better 
account  for  this  to  make  the  appraisal  more  efficient.  As  more  information  about  this 
bifurcation  of  practices  was  learned  in  the  three  later  mini-appraisals,  these  PIIs 
continued  to  evolve. 

3.  As  TCS  has  many  projects  at  each  location,  all  projects  could  not  be  included.  A  sample 
demographic  representation  was  taken  to  ensure  that  all  units  were  covered  sufficiently. 

People  CMM  has  a  three-tier  structural  view  of  an  organization:  (1)  the  organization 
itself,  (2)  units  within  the  organization  which  are  represented  by  the  managers  in  the 
organization,  and  (3)  workgroups  (or  projects)  and  individuals  who  are  represented  as 
the  workforce.  While  unit  representation  was  assured  during  the  first  pilot,  many  more 
people  were  interviewed  than  might  be  required  for  a  SCAMPI  with  CMMI. 

4.  The  organization  view  is  represented  through  the  process  owners  and  others  who  are 
responsible  at  the  corporate  and  center  levels  above  the  project  level.  These  individuals 
can  best  help  the  Class  A  appraisal  by  being  interviewed  early  in  the  appraisal  to  provide 
the  overall  organizational  process  perspectives.  This  input  aids  the  contextual 
understanding  for  the  unit  and  individual  interviews  to  follow. 

The  process  results  from  the  mini-appraisals  enabled  a  better  understanding  of  how  to 
perform  the  Class  A  appraisal.  For  future  Class  A  appraisals,  the  completion  of  Class  C 
and/or  B  appraisals  could  also  be  considered  to  provide  understanding  before  the  Class  A  is 
performed. 
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3.2  Lessons  Learned  from  the  Enterprise  Appraisal 

The  objective  after  the  four  mini-appraisal  pilots  was  to  perform  an  enterprise  appraisal 
across  all  of  TCS.  This  accounted  for  approximately  29,000  individuals.  This  was  a  large 
appraisal  and  was  covered  over  a  six  and  a  half  month  period  during  2004.  Not  all  the  time 
was  spent  in  active  appraising  activities,  as  the  appraisal  team  was  given  two-week  breaks 
between  major  planned  activities. 

The  enterprise  appraisal  used  v0.7  and  v0.8  of  this  document  as  the  base  SCAMPI  with 
People  CMM  definition.  This  enterprise  appraisal,  while  large,  was  made  easier  by  the 
history  of  TCS  with  16  CMM  Level  5  and  four  People  CMM  Level  4  locations  worldwide 
prior  to  the  start  of  the  mini-appraisals. 

The  information  in  this  report  could  serve  as  an  example  for  other  organizations  and  is  fully 
applicable  to  any  size  of  SCAMPI  appraisal  with  People  CMM.  (See  Appendix  I  for 
additional  lessons  learned  regarding  a  combined  People  CMM  and  CMMI  appraisal.)  Also  as 
a  result  of  this  pilot,  the  software  community  can  use  the  knowledge  gained  from  the 
development  of  a  SCAMPI  with  People  CMM  appraisal  method. 
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Appendix  A  Sampling 


Sampling  is  performed  when  a  population  is  too  large  or  the  cost  of  obtaining  information 
from  the  full  population  is  too  high. 

Since  the  TCS  class  appraisal  was  for  an  organization  of  more  than  29,000  people,  sampling 
was  required  and  was  performed  for  project  and  interview  participants  to  ensure  the 
following: 

•  random  selection 

•  representative  selection 

•  non-biased  selection 

Random  Sampling— Random  selection  was  used  so  that  any  individual  in  the  workforce, 
manager,  or  unit  would  be  as  likely  to  be  selected  as  any  other. 

Representative  Sampling — The  selected  sampled  subset  was  demonstrated  to  be 
representative  of  the  organization  (i.e.,  the  sampling  technique  did  not  inadvertently  cause  a 
demographic  misrepresentation  of  the  organization).  For  example,  a  significant  unit  was  not 
included  above  any  other  unit  and  no  age  groups  were  excluded. 

Non-Biased  After  Selection — Although  the  focus  was  on  selecting  a  random  and 
representative  sample,  the  sample  received  an  independent  review  (outside  of  the  appraisal 
team)  to  ensure  that  no  biases  were  introduced. 

Selecting  Samples — Samples  were  selected  as  close  as  possible  to  the  time  of  engagement 
because  a  sample  selected  too  early  might  not  have  been  representative  of  the  organization’s 
views  at  the  time  of  the  appraisal. 
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Appendix  B  50%  Rule 


Satisfaction  of  the  50%  rule  is  a  minimum  requirement  for  each  goal.  Satisfaction  of  the  rule 
demonstrates  coverage  of  practices  and  sufficiency  of  performance  across  units  relevant  to 
the  practice.  The  rule,  simply  stated,  requires  that  50%  of  the  process  area  practices  for  each 
goal  in  each  unit  in  the  set  of  units  to  be  addressed  within  the  appraisal  have  at  least  one  face- 
to-face  affirmation  data  point. 

This  leads  to  what  is  sometimes  called  the  “one  row-one  column”  rule.  In  Figure  1  below,  we 
see  that  goal  X  has  four  related  practices.  The  organization  being  appraised  is  a  small 
organization  with  only  four  units. 


Figure  1:  Satisfaction  Rules 


In  this  example,  according  to  the  50%  rule,  each  practice  must  have  face-to-face  affirmations 
from  at  least  two  units  since  there  are  four  units  in  this  appraisal. 

The  one  row-one  column  rule  means  that  for  each  practice  and  each  unit  there  will  be  at  least 
one  checkmark  in  the  practice-unit  matrix  coordinates.  This  will  prevent  some  units  or  some 
practices  from  having  no  face-to-face  affirmations. 
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For  a  small  organization,  it  is  easy  to  view  all  the  columns  in  Figure  1;  however,  large, 
complex  organizations  require  many  units,  and  the  matrix  can  become  complicated.  A 
spreadsheet  is  needed  to  ensure  that  data  is  captured  and  kept  visible  to  the  team  during  the 
appraisal.  The  ATL  must  track  face-to  face  affirmations  to  make  sure  they  are  completed. 

The  50%  rule  and  the  one  row-one  column  rule  are  needed  to  ensure  that  the  appraisal  does 
not  become  paper  focused.  Recall  that  the  rules  for  direct  artifacts  with  indirect  artifacts 
prove  corroboration.  However,  if  only  these  artifacts  are  used,  the  reason  for  getting 
affirmations  from  interviews  is  lost.  Thus,  with  these  two  rules,  the  method  assures  that 
interviews  demonstrate  that  an  organization  is  performing  the  model  practices. 

Some  practices  are  performed  at  the  organization  level  and  some  at  the  unit  level.  For  those 
performed  at  the  unit  level,  all  units  sampled  as  representative  of  the  organization  are 
included  in  the  measure  of  the  50%  rule.  However,  when  practices  are  performed  at  the 
organization  level  the  rules  still  apply. 

How  to  Score  Practices  Stated  “As  Appropriate”  in  People  CMM 

Some  practices  in  the  People  CMM  include  words  such  “as  appropriate”  or  “where 
appropriate.”  For  these  practices  there  are  three  possible  situations  during  the  appraisal: 

1 .  There  are  artifacts  and  an  affirmation  stating  that  the  practice  is  being  performed  as 
required  by  the  model.  In  this  case,  the  “4”  is  warranted  in  the  unit/practice  matrix  for 
the  relevant  unit  and  practice. 

2.  There  are  no  artifacts  demonstrating  that  the  practice  is  being  performed  as  required  by 
the  model  and/or  there  is  no  affirmation  stating  the  practice  is  being  performed.  In  this 
case,  the  “4”  is  not  warranted  in  the  unit/practice  matrix  for  the  relevant  unit  and 

.  practice. 

3.  There  are  no  artifacts  demonstrating  that  the  practice  is  being  performed  as  required  by 
the  model  and/or  there  is  no  affirmation  stating  the  practice  is  being  performed,  and 
there  is  a  clear  interpretation  that  the  practice  is  not  appropriate  for  the  unit.  In  this 
case,  the  “4”  is  given  as  an  equivalent  for  scoring  purposes  only  in  the  unit/practice 
matrix  for  the  relevant  unit  and  practice,  and  the  practice  is  not  applicable  (NA) 

for  this  unit. 
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Appendix  C  Indirect  Artifacts 


The  practice  implementation  indicators  (PEs)  define  generic  direct  and  indirect  artifacts  for 
each  practice  in  People  CMM.  Generic  PEs  are  provided  apart  from  the  interpretation  guide 
in  Section  2. 

Specific  direct  artifacts  were  found  for  all  practices  during  the  SCAMPI  with  People  CMM 
pilot.  However,  since  the  People  CMM  did  not  intend  to  have  SCAMPI  applied  when  written, 
it  is  possible  that  some  practices  in  some  organizations  might  not  have  indirect  artifacts. 

In  the  TCS  mini-appraisals  and  the  enterprise  appraisal,  direct  artifacts  were  found  for  each 
practice.  Indirect  artifacts  were  seen  for  all  practices,  but  not  in  all  entities.  Because  the 
greatest  focus  was  on  interview  evidence,  indirect  artifacts  were  not  always  pursued.  While 
some  organizations  might  not  have  an  indirect  artifact  for  a  practice,  these  instances  should 
be  relatively  few.  As  noted  in  activity  2. 1.4.5  Obtain  Initial  Objective  Evidence,  some 
practices  might  not  have  indirect  artifacts  in  all  organizational  implementations  of  the 
practice. 

Because  SCAMPI  permits  the  50%  rule,  indirect  evidence  is  required  only  for  50%  of  the 
entities  when  50%  of  the  evidence  is  from  interviews.  When  an  appraisal  focuses  more 
heavily  on  interview  evidence,  the  need  for  indirect  artifacts  is  reduced  or  nonexistent  when 
the  interviews  supply  sufficient  evidence.  Therefore,  when  a  SCAMPI  with  People  CMM  is 
planned,  the  plan  should  address  the  practices  that  might  not  have  indirect  artifacts. 

Interviews  will  become  a  deciding  factor  for  practice  compliance  and  goal  satisfaction. 

The  version  of  the  interpretation  guide  in  Section  2  does  not  list  all  potential  practices  that 
might  not  have  indirect  artifacts.  However,  two  practices  that  might  not  have  indirect  artifacts 
are  listed  below. 

•  Performance  Management — Goal  2:  P7:  Those  responsible  for  managing  the 
performance  of  others  maintain  an  awareness  of  accomplishments  against  performance 
objectives  for  each  of  the  individuals. 

•  Continuous  Capability  Improvement — Goal  2:  P3:  Individuals  evaluate  the  capability  of 
their  personal  work  processes  to  identify  opportunities  for  improvement. 
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Appendix  D  Comparisons  of  People 

CMM-BAM  Description  and 
Interpreting  SCAMPI  for  a 
People  CMM  Appraisal 


Table  3:  Map  1  of  Interpreting  SCAMPI  for  a  People  CMM  Appraisal  to  People 
CMM  BAM  Description  V1.0 


i  Interpreting  SCAMPI  for  a  People  CMM 
Appraisal  (This  Document) 

People  CMM  BAM  Description  V  1.0 

2.1.1  Analyze  Requirements 

2.1.1.1  Determine  Appraisal  Objectives 

2.2.1.1  PI  Secure  Improvement  Sponsor 

2.1.1.2  Determine  Appraisal  Constraints 

2.2.1.2  P2  Determine  Assessment  Scope 

2.1.13  Determine  Appraisal  Scope 

2.2.1.2  P2  Determine  Assessment  Scope 

2.1.1.4  Determine  Outputs 

2.2.1.3  P3  Obtain  Organization  Commitment 

2.1.1.5  Obtain  Commitment  to  Appraisal  Input 

2.2.1.1  PI  Secure  Improvement  Sponsor 

2.1.2  Develop  Appraisal  Plan 

2.1.2.1  Tailor  Method 

2.2.1.5  P5  Plan  Assessment  Details 

2.1. 2.2  Identify  Needed  Resources 

2.2.1.5  P5  Plan  Assessment  Details 

2.1.2.3  Determine  Cost  and  Schedule 

2.2.1.5  P5  Plan  Assessment  Details 

2.1.2.4  Plan  and  Manage  Logistics 

2.2.1.4  P4  Arrange  Assessment  Logistics 

2.1.2.5  Document  and  Manage  Risks 

2.2.1.5  P5  Plan  Assessment  Details 

2.1.2.6  Obtain  Commitment  to  Appraisal  Plan 

2.2.1.3  P3  Obtain  Organization  Commitment 

2.1.3  Select  and  Prepare  Team 

2.1.3.1  Identify  Team  Leader 

2.2.3.1  A1  Organize  Assessment  Team 

2.1.3.2  Select  Team  Members 

2.2.3.1  A1  Organize  Assessment  Team 

2. 1.3.3  Prepare  Team 

2.2.1.6  P6  Train  Assessment  Team 

2.1.4  Obtain  and  Analyze  Initial  Objective  Evidence 
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Interpreting  SCAMPI  for  a  People  CMM 
Appraisal  (This  Document) 

2.1.4.2  Prepare  Participants 


2.1. 4.3  Administer  Instruments 


2.1.4.4  Obtain  Initial  Objective  Evidence 


2.1.4.5  Inventory  Objective  Evidence 


2.1.5  Prepare  for  Collection  of  Objective  Evidence 


2.1.5.1  Perform  Readiness  Review 


2.1.5.2  Prepare  Data  Collection  Plan 


People  CMM  BAM  Description  Vl.O 


2.2.3.2  A2  Brief  Assessment  Participants 


2.2.2 3  S3  Administer  and  Score  People  CMM 
Surveys 


2.2.1.4  P4  Arrange  Assessment  Logistics 


2.2.3.4  A4  Review  Documents 


2.2.3.4  A4  Review  Documents 


2.2.1.5  P5  Plan  Assessment  Details 
2.2.3.5A5  Script  Interviews 


2.2.1  Examine  Objective  Evidence 


2.2.1.1  Examine  Objective  Evidence  from  Instruments  2.2.2.4  S4  Analyze  People  CMM  Survey  Results 


2.2.1.2  Examine  Objective  Evidence  from 
Presentations 


2.2.3.4  A4  Review  Documents 


2.2.1.3  Examine  Objective  Evidence  from  Documents  2.2.3.4  A4  Review  Documents 


2.2.1.4  Examine  Objective  Evidence  from  Interviews 


2.2.3.6  A6  Interview  Process  Owners 

2.2.3.9  A9  Interview  Managers 

2.2.3.11  All  Conduct  Workforce  Discussions 


2.2.2  Verify  and  Validate  Objective  Evidence 


2.2.2.1  Verify  Objective  Evidence 


2.2.2.2  Characterize  Implementation  of  Model  Practices 


2.2.2.3  Validate  Practice  Implementation  Gaps 


2.2.3.7  A7  Consolidate  Process  Owner  Data 
2.23.8  A8  Perform  Follow-up  Interviews 
2.2.3.10  A10  Consolidate  Manager  Data 
2.2.3.12  A12  Consolidate  Workforce  Data 


2.23.13  A13  Develop  Preliminary  Assessment 
Findings 


2.2.3.15  A15  Review  Preliminary  Findings  with 
Legal 

2.2.3.16  A16  Review  Preliminary  Findings  with 
Process  Owners 

2.2.3.17  A17  Review  Preliminary  Findings  with 
Workforce 

2.2.3.18  A18  Review  Preliminary  Findings  with 
Managers 


2.2.3  Document  Objective  Evidence 


2.23.1  Take/Review/Tag  Notes 


2.23.2  Record  Presence/ Absence  of  Objective 
Evidence 


2.23.4  A4  Review  Documents 


2.2.3.14  A14  Prepare  Preliminary  Findings  Briefing 
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Table  4:  Map  2  of  Interpreting  SCAMPI  for  a  People  CMM  Appraisal  to  People 
CMM  BAM  Description  VI. 0 


Interpreting  SCAMPI  for  a  People  CMM  People  CMM  BAM  Description  V1.0 
Appraisal  (This Document)  •  •  r-.  ", 

2.2.1  Preparing  Phase 


2.1.1.1  Determine  Appraisal  Objectives 
2.1.1.5  Obtain  Commitment  to  Appraisal  Input 


2.1.1.2  Determine  Appraisal  Constraints 

2.1.1.3  Determine  Appraisal  Scope 


2.1.1.4  Determine  Outputs 

2.1.2.6  Obtain  Commitment  to  Appraisal  Plan 


2.1.2.4  Plan  and  Manage  Logistics 


2.1.2.1  Tailor  Method 

2.1.2.2  Identify  Needed  Resources 

2.1.2.3  Determine  Cost  and  Schedule 

2.1.2.5  Document  and  Manage  Risks 

2.1.5.2  Prepare  Data  Collection  Plan 


2.1.3.3  Prepare  Team 


2.1.2.4  Plan  and  Manage  Logistics 

2.1.4.3  Obtain  Initial  Objective  Evidence 


2.2.1.1  PI  Secure  Improvement  Sponsor 


2.2.1.2  P2  Determine  Assessment  Scope 


2.2.1.3  P3  Obtain  Organizational  Commitment 


2.2.1.4  P4  Define  Improvement  Infrastructure 

2.2.1.5  PI  Plan  Assessment  Details 


2.2.1.6  P6  Train  Assessment  Team 


2.2.1.7  P7  Arrange  Assessment  Logistics 


2.2.2  Surveying  Phase 


2.2.2.1  SI  Select  People  CMM  Survey  Sample 

2.2.2.2  S2  Prepare  Survey  Logistics 


2.2.2.3  S3  Administer  and  Score  People  CMM 

2.1.4.2  Administer  Instruments  Surveys _ 

2.2.1.1  Examine  Objective  Evidence  from  Instruments  2.2.2.4  S4  Analyze  People  CMM  Survey  Results 

2.2.3  Assessing  Phase 


2.1.3.1  Identify  Team  Leader 

2.1.3.2  Select  Team  Members 


2.1.4.1  Prepare  Participants 


2.2.1.1  Examine  Objective  Evidence  from  Instruments 


2.2.3.1  A1  Organize  Assessment  7  earn 


2.2.3.2  A2  Brief  Assessment  Participants 


2.233  A3  Analyze  People  CMM  Survey  Results 


2.1.4.4  Inventory  Objective  Evidence 

2.1.5.1  Perform  Readiness  Review 

2.2.1.2  Examine  Objective  Evidence  from  Presentations 
2.2.13  Examine  Objective  Evidence  from  Documents 


2.23.4  A4  Review  Documents 


2.23.1  Take/Review/Tag  Notes 
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Interpreting  SCAMPI  for  a  People  CMM 
Appraisal  (This  Document) 


2.1.5.2  Prepare  Data  Collection  Plan 


2.2.1.4  Examine  Objective  Evidence  from  Interviews 
2.2.3.1  Take/Review/Tag  Notes 


2.2.2.1  Verify  Objective  Evidence 


People  CMM  BAM  Description  VI 


2.2.3.S  AS  Script  Interviews 


2.2.3.6  A6  Interview  Process  Owners 


2.2.3.7  A7  Consolidate  Process  Owner  Data 


2.2.3.4  Review  and  Update  the  Data  Collection  Plan  2.2.3.8  A8  Perform  Follow-up  Interviews 


2.2.1.4  Examine  Objective  Evidence  from  Interviews 
2.2.3.1  Take/Review/Tag  Notes 


2.2.2.1  Verify  Objective  Evidence 


2.2.1.4  Examine  Objective  Evidence  from  Interviews 

2.2.3.1  Take/Review/Tag  Notes 


2.2.2.1  Verify  Objective  Evidence 


2.2.2.2  Characterize  Implementation  of  Model  Practices 


2.2.3.2  Record  Presence/ Absence  of  Objective 
Evidence 

2.2.3.3  Document  Practice  Implementation  Gaps 


2.2.2.3  Validate  Practice  Implementation  Gaps 


2.2.2.3  Validate  Practice  Implementation  Gaps 


2.2.2.3  Validate  Practice  Implementation  Gaps 


2.2.2.3  Validate  Practice  Implementation  Gaps 


2.2.4.1  Derive  Findings  and  Rate  Goals 
2.2.4.2b  Determine  Satisfaction  of  Process  Areas 
2.2.4.3b  Determine  Maturity  Level 


2.2.4.4  Document  Appraisal  Results 


2.3.1.1  Present  Final  Findings 


2.3.1.2  Conduct  Executive  Session(s) 


2.3.1.3  Plan  for  Next  Steps 
2.3.2.1  Collect  Lessons  Learned 

2.3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


2.2.3.9  A9  Interview  Managers 


2.2.3.10  A10  Consolidate  Manger  Data 


2.2.3.11  All  Conduct  Workforce  Discussions 


2.2,3.12  A12  Consolidate  Workforce  Data 


2.2.3.13  A13  Develop  Preliminary  Assessment 
Findings 


2.2.3.14  A14  Prepare  Preliminary  Findings 
Briefing 


2.2.3.15  A15  Review  Preliminary  Findings  with 
Legal  Representative 


2.2.3.16  A16  Review  Preliminary  Findings  with 
Process  Owners 


2.2.3.17  A7  Review  Preliminary  Findings  with 
Workforce 


2.2.3.18  A18  Review  Preliminary  Findings  with 
Managers 


2.2.3.19  A19  Revise  Final  Findings  and  Rate 
Maturity 


2.2.3.20  A20  Prepare  Final  Findings  Briefing 


2.2.3.21  A21  Present  Final  Findings 


2.2.3.22  A22  Debrief  Sponsor 


2.2.3.23  A23  Wrap-up  Assessment 


2.2.4  Reporting  Phase 
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Appendix  E  Organizational  Structures 


As  defined  in  People  CMM,  “unit,”  has  recursive  meanings.  A  unit  is  a  single,  well-defined 
organizational  component  (e.g.,  a  department,  section,  or  project)  of  an  organization.  The 
term  “unit”  is  used  to  refer  to  any  organizational  entity  that  is  accountable  to  a  specified 
individual(s)  (usually  a  manager)  responsible  for  accomplishing  a  set  of  performance 
objectives  that  can  be  met  only  through  collective  action.  A  workgroup  may  constitute  the 
lowest-level  unit,  but  the  lowest-level  units  often  consist  of  several  workgroups.  “Unit”  is  a 
recursive  concept;  units  may  be  composed  from  other  units  cascading  down  the  organization. 
For  instance,  a  division  may  be  a  unit  consisting  of  departments,  each  of  which  may  be  a  unit 
consisting  of  programs,  each  of  which  may  be  a  unit  consisting  of  projects,  and  so  on. 

To  be  consistent  with  the  SCAMPI,  project  units  are  selected  as  entities  for  an  appraisal.  The 
projects  are  randomly  selected  and  demographically  represent  the  organization  in  order  for 
the  appraisal  to  be  considered  valid.  There  must  be  sufficient  representation  when  selecting 
projects  for  the  appraisal.  Appendix  A  explains  how  to  make  a  sufficient  and  representative 
selection. 

“Workgroup”  is  used  the  same  way  as  in  People  CMM.  A  workgroup  might  or  might  not  be  a 
“project.”  The  term  “organization”  can  also  be  recursive.  An  organization  is  defined  as  a  utiit 
within  a  company  or  other  entity  (e.g.,  government  agency  or  branch  of  service)  within  which 
many  projects  are  managed  as  a  whole.  All  projects  within  an  organization  share  a  common 
top-level  manager  and  common  policies. 

Any  term  that  clusters  projects/units  might  be  the  appraised  organization,  but  the  scope  is 
limited  to  the  definition  of  organization  in  the  appraisal  plan.  An  appraised  organization 
might  have  multiple  sub-levels  that  all  act  above  the  project/unit  level.  For  the  purposes  of 
the  appraisal  these  would  be  “organization”  entities,  and  when  practices  are  determined  to  be 
organizational  in  focus,  the  appraisal  team  must  determine  which  level  of  “organization” 
should  be  interviewed  and  supply  artifacts  for  each  related  practice. 

The  practices  within  the  People  CMM  should  be  mapped  by  the  appraisal  team  using  the 
generic  PIIs  to  allocate  each  practice  to  project/unit  or  organization,  including  the  level 
of  the  organization. 
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Appendix  F  Appraisal  Disclosure 

Statement  (ADS) 


The  appraisal  disclosure  statement  (ADS)  provides  information  essential  for  the  adequate 
interpretation  of  the  maturity  level  or  capability  level  ratings  resulting  from  a  People  CMM 
Class  A  appraisal. 

The  ADS  is  prepared  by  the  appraisal  team  leader  and  provided  to  the  appraisal  sponsor  at  the 
conclusion  of  the  appraisal.  If  the  final  findings  briefing  reports  the  appraisal  ratings,  the 
vehicle  for  reporting  the  ratings  must  be  the  ADS.  Otherwise  the  appraisal  team  leader 
delivers  the  ADS  to  the  sponsor  as  a  separate  document. 

The  ADS  consists  of  the  following  information: 

•  identification  of  appraisal  sponsor  and  sponsor’s  organizational  affiliation 

•  identification  of  appraisal  team  leader  and  appraisal  team  members  and  their 
organizational  affiliations 

•  identification  of  organizational  unit  appraised  (i.e.,  the  unit  to  which  the  ratings  are 
applicable  and  the  domains  examined,  as  defined  in  the  appraisal  plan) 

•  identification  of  reference  model(s)  (version,  representation,  and  domains) 

•  identification  of  appraisal  method  (name  and  version) 

•  itemization  of  process  areas  rated  and  process  areas  not  rated 

•  maturity  level  and/or  capability  level  ratings  assigned 

•  dates  of  on-site  activity 

•  date  of  issuance  of  ADS 

•  statement  affirming  that  all  SCAMPI  requirements  were  met 

•  signature  of  appraisal  team  leader  (at  a  minimum);  signatures  of  appraisal  team  members 
and  appraisal  sponsor  (optional) 
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Appendix  G  The  Role  of  Practice 

Implementation  Indicators 


This  appendix  provides  a  conceptual  overview  of  the  process  used  to  verify  practice 
implementation  and  the  role  of  practice  implementation  indicators.  Verifying  the 
implementation  of  processes  relative  to  models  of  best  practices  is  essential. 

Verifying  Practice  Implementation 

Verifying  that  People  CMM  practices  are  implemented  requires  a  review  based  on  objective 
evidence.  This  would  include  such  things  as  verifying  that  a  unit-specific  practice  is 
implemented  within  a  unit  or  an  organization-specific  practice  is  implemented  within  an 
organization. 

Having  a  well-defined  approach  for  verifying  practice  implementation  is  critically  important 
for  several  reasons.  For  the  process  improvement  sponsor,  it  provides  some  assurance  that  the 
resources  applied  to  the  improvement  effort  will  result  in  the  desired  outcome  and  benefits. 
For  process  improvement  agents  or  champions,  it  enables  them  to  know  when  they  have 
succeeded  with  the  implementation  activity  and  to  informally  monitor  whether  the  practice 
continues  to  be  implemented  over  time. 

For  appraisal  teams,  it  allows  them  to  determine  what  capability  level  or  maturity  level 
ratings  are  warranted.  People  CMM  process  area  goal  satisfaction  is  predicated  on 
implementation  of  the  relevant  specific  or  institutionalization  practices  or  acceptable 
alternatives.  Hence  verification  of  practice  implementation  is  a  crucial  appraisal  task.  (See 
Section  4.6  of  the  People  CMM:  Required,  Expected,  and  Informative  Components.) 

Practice  Implementation  Indicators 

The  use  of  practice  implementation  indicators  (PIIs)  is  simple  and  applicable  to  any  practice 
or  activity.  It  is  based  on  the  idea  that  the  conduct  of  an  activity  or  the  implementation  of  a 
practice  will  result  in  “footprints” — evidence  that  the  activity  was  conducted  or  the  practice 
was  implemented.  For  example,  if  one  balances  a  checkbook  at  the  end  of  the  month,  there 
are  several  potential  ways  to  confirm  that  this  activity  has  taken  place.  First,  the  person  who 
balanced  it  can  affirm  that  this  activity  was  conducted.  Second,  there  will  likely  be  an  entry 
in  the  checkbook  register  for  each  check  or  transaction  to  indicate  that  it  matches  with  a 
corresponding  entry  on  the  bank  statement. 
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PIIs  refer  to  the  footprints  that  are  the  necessary  and  unavoidable  consequence  of  practice 
implementation.  They  include  information  contained  in  artifacts  and  information  gathered 
from  interviews  with  managers  and  practitioners  and  with  the  People  CMM  survey,  if  used. 


The  Role  of  PIIs 

ARC-compliant  appraisal  methods  employ  objective  evidence  obtained  from  one  or  more 
sources  (e.g.,  instruments,  documents,  and/or  interviews).  An  appraisal  team  bases  its 
decisions  about  practice  implementation  on  examination  of  this  objective  evidence. 

Once  an  organization  has  an  understanding  of  how  its  processes  relate  to  the  People  CMM 
model,  it  is  possible  to  capture  PIIs  that  provide  objective  evidence  of  implementation. 
Establishing  the  collection  of  PIIs  for  the  organization  provides  assurance  to  the  process 
improvement  sponsor  that  the  expected  implementation  activities  have  resulted  in  the 
alignment  of  the  organization’s  activities  with  the  People  CMM. 

This  aggregation  of  objective  evidence — the  PIIs — is  an  important  organizational  process 
asset  with  a  number  of  potential  uses,  most  notably  providing  the  appraisal  team  with  a  head 
start  in  understanding  the  organization’s  implementation  of  the  People  CMM.  This  allows  the 
appraisal  team  to  verify  that  the  objective  evidence  provided  is  adequate  for  substantiation  of 
practice  implementation  rather  than  having  to  perform  the  more  difficult,  error-prone,  and 
time-consuming  task  of  investigating  each  practice  to  find  the  objective  evidence  needed  to 
substantiate  implementation. 

The  appraised  organization  and  the  appraisal  team  have  a  clearer  picture  of  what  artifacts 
need  to  be  provided  to  substantiate  implementation  of  the  practices,  minimizing  the  number 
of  further  interviews  and  documentation  requests  required.  The  extent  to  which  the  appraised 
organization  can  provide  this  information  helps  determine  how  much  further  investigation  is 
required. 

This  approach  also  provides  a  significantly  more  reliable  and  accurate  appraisal.  The  PII- 
based  approach  is  not  meant  to  turn  the  appraisal  into  a  documentation  review  exercise.  It 
merely  allows  for  a  more  focused,  effective,  and  possibly  shorter  on-site  phase. 

Finally,  the  PIIs  are  not  intended  to  tie  the  hands  of  model  implementers  or  process  appraisal 
teams.  The  primary  value  of  the  PIIs  lies  in  making  explicit  what  has  been  subject  to  wide 
variations  in  interpretation  and  understanding.  Over  time,  sharing  PIIs  will  result  in  a  set  of 
practice  implementation  scenarios  (e.g.,  for  small,  medium,  and  large  organizations  or  units) 
and  a  standard  set  of  PIIs  that  could  be  used  as  a  starting  point  for  further  customization.  The 
particular  process  implementation  context  and  the  unit  specifics  would  determine  which 
indicators  make  sense  for  that  implementation.  Appraisal  teams  would  be  able  to  find  out  if 
•  the  agreed-upon  indicators  exist  while  still  having  the  freedom  to  make  judgments  based  on 
the  facts  and  circumstances  of  the  implementation. 
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A  standard  set  of  PIIs  could  establish  the  norms  within  which  most  implementations  would 
fall,  allowing  implementation  and  appraisal  efficiencies  to  be  realized  while  at  the  same  time 
recognizing  that  alternative  implementations  using  alternative  practices  might  be  possible. 

Pll  Components 

PIIs  have  two  components  or  dimensions:  an  objective  evidence  component  and  a  practice 
implementation  component.  The  objective  evidence  component  is  the  form  of  the  objective 
evidence.  The  practice  implementation  is  the  significance  of  the  objective  evidence  in 
relation  to  the  practice  implementation. 

Forms  of  Objective  Evidence 

An  appraisal  team  bases  its  practice  implementation  decisions  on  the  objective  evidence 
available,  which  can  take  on  one  or  both  of  the  following  forms: 

•  artifacts 

-  work  products,  which  are  the  explicit  intended  consequences  of  practice 
implementation 

-  artifacts  that  are  incidental  to,  but  indicative  of,  practice  implementation 

•  affirmations 

-  written  or  oral  statements  from  practitioners  who  carry  out  the  activities  relevant  to 
the  practice  or  from  other  stakeholders  in  the  practice 

-  demonstrations  or  presentations  (e.g.,  the  demonstration  of  the  capability  of  a  tool  or 
other  mechanism  as  it  relates  to  the  implementation  of  a  practice  or  a  presentation 
explaining  some  aspect  of  the  organization  or  project) 

Note  that  there  is  not  a  strong  distinction  in  the  model  between  artifacts  and  work  products. 
In  the  context  of  PIIs,  “work  product”  refers  to  an  artifact  that  is  either  explicitly  mentioned 
in  the  statement  of  a  People  CMM  practice  or  whose  absence  would  be  a  strong  indicator  of 
incomplete  or  inadequate  practice  implementation.  The  weaker  term  “artifact”  is  used  in  the 
context  of  PIIs  to  refer  to  an  artifact  whose  existence  is  incidental  to  the  accomplishment  of 
the  practice  (i.e.,  a  side-effect). 

Types  of  PIIs 

Using  the  above  discussion  as  the  framework,  it  is  possible  to  itemize  the  types  of  PIIs  that 
might  be  present  as  a  consequence  of  practice  implementation.  Table  5  shows  PII  types, 
which  collectively  provide  coverage  for  any  People  CMM  practice.  Each  type  is  described  in 
more  detail  in  the  pages  that  follow. 
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Table  5:  Pll  Types 


PII 

Type 

Objective 

Evidence  Form 

Generic  Description 

Direct 

Artifact  (work  product) 

Work  produces)  that  reflect  (document  the  information 
content  of)  the  establishment  of  { insert  text  from  practice 
statement  that  describes  object  of  practice  enactment}. 

Indirect 

Artifact 

Artifact(s)  that  are  an  indirect  consequence  (or  side-effect) 
of  the  effort  required  to  {insert  text  from  practice  statement 
that  describes  object  of  practice  enactment}. 

Direct 

Affirmation 

Affirmations  from  individuals  who  participated  in  or 
contributed  to  {insert  text  from  practice  statement  that 
describes  object  of  practice  enactment}  OR  affirmations 
from  individuals  who  are  users  of  (or  who  can  substantiate 
use  of)  {insert  text  from  practice  statement  that  describes 
object  of  practice  enactment}. 

Direct  Artifact — This  PH  type  is  relevant  when  the  establishment  of  a  work  product  is  an 
integral  part  of  practice  implementation.  Sometimes  this  is  explicit,  as  in  Competency 
Analysis  Practice  7  which  says,  “Competency  information  regarding  the  capabilities  of 
individuals  in  their  workforce  competencies  is  collected  and  maintained  according  to  a 
documented  procedure.”  In  other  instances  it  is  not  explicit,  although  it  would  be  difficult  to 
imagine  practice  implementation  without  the  presence  of  one  or  more  work  products.  In  most 
cases,  the  model  document  already  identifies  these  work  products. 

Indirect  Artifact— This  PH  type  applies  to  artifacts  that  are  produced  as  a  natural 
consequence  of  the  enactment  of  a  practice.  In  contrast  to  a  direct  artifact  Pll.  this  type  of 
artifact  is  an  indirect  consequence  or  side-effect  of  practice  enactment.  Indirect  artifacts  that 
are  relevant  to  this  PII  will  vary  widely  and  will  tend  to  be  implementation  specific  This 
indicator  type  is  especially  useful  if  there  are  doubts  about  whether  the  intent  of  the  practice 
has  been  met  (e.g.,  a  work  product  exists  but  there  is  no  indication  of  where  it  came  from  or 
who  developed  it). 

Direct  Affirmation — This  PII  type  refers  to  information  obtained  though  interviews  of  the 
individuals  who  are  involved  in  or  stakeholders  of  the  enactment  of  a  practice.  This  type  can 
also  apply  to  information  provided  in  other  ways,  such  as  through  demonstrations  or 
presentations. 
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Pll  Descriptions 

A  PII  description  (PHD)  is  a  structure  or  schema  defined  to  provide  a  repository  for  the  PII 
information.  Table  6  shows  an  example  of  such  a  structure.  Note  that  this  is  a  notional 
description  of  the  content,  not  a  physical  definition  of  the  format. 

Table  6:  A  PIID  Schema  for  People  CMM 


Attribute 

Synopsis 

Remarks 

Practice  ID 

This  identifies  the  process  area,  goal,  and  practice 
that  the  PII  is  associated  with. 

Acronyms  found  in  the  People 
CMM  model  are  used. 

PII  ID 

This  identifies  the  indicator  type  and  the  form  of 
objective  evidence 

Types  are  direct  artifact, 
indirect  artifact,  and  direct 
affirmation. 

Description 

This  is  a  description  of  the  PII  as  applied  to  this 
practice. 

Examples 

These  are  examples  of  artifacts  or  affirmations 
that  would  exemplify  the  intent  of  the  PII  and/or 
exploratory  questions  (EQs)  or  "look  fors"  (LFs). 
They  assist  appraisers  in  identifying  relevant 
artifacts  or  eliciting  relevant  information. 

Aim  to  minimize  any  overlap 
with  such  information  that  is— 
already  in  the  model  document. 

Organizational 

Implementation 

This  attribute  would  be  filled  in  by  the 
organization  as  part  of  its  implementation 
program  and  provided  to  the  appraisal  team  as  a 
resource. 

Table  7  shows  an  example  PHD  for  specific  practice  1.1-1  of  the  People  CMM  staffing 
process  area. 

Table  7:  An  Example  PIID  for  People  CMM 


Attribute 

'Value 

Practice  ID 

STFP1 

PII  ID 

Direct  Artifact 

Description 

Plans  that  reflect  and  coordinate  the  staffing  activities  in  a  project  (unit)  in 
accordance  with  documented  organization  policies  and  procedures 

Examples 

Project  staffing  plan 

Organizational 

Implementation 

Staffing  plans  at  the  organization  level. 

Policies  and  procedures  at  the  organizational  level  for  all  projects. 
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These  descriptions  have  a  number  of  uses  beyond  process  appraisals.  For  example,  they  can 
be  used  during  the  model  implementation  phase,  after  model  implementation  as  a  training 
vehicle  for  new  personnel,  or  for  internal  monitoring  of  practice  implementation. 

Application  of  Plls  in  Model  Implementation 

Indicator  use  benefits  organizations  committed  to  model-based  process  improvement. 
Typically,  organizations  will  either  implement  model  practices  directly  or  will  ensure  that  the 
practices  used  in  the  organization  help  to  achieve  goals  (through  the  mechanism  of  alternative 
practices). 

Since  models  have  to  be  expressed  and  published  in  an  implementation-independent  manner, 
the  model  implementation  requires  an  understanding  of  how  the  model  intent  (as  expressed 
though  goals,  practices,  and  other  model  material)  is  to  be  realized  in  the  organization.  The 
model  intent  is  made  real  through  its  impact  on  the  way  people  work:  if  there  is  no  relation 
between  how  they  work  and  the  model,  the  organization  has  not  implemented  the  model. 
Thus,  understanding  how  the  implementation  of  the  model  relates  to  what  people  are  doing  in 
the  organization  is  a  necessary  and  unavoidable  prerequisite  to  implementing  the  model. 

PTTDs  provide  a  mechanism  for  describing  the  implementation  of  a  model  practice. 

Application  of  Plls  in  Process  Appraisal 

During  the  course  of  a  process  appraisal,  the  appraisal  team’s  primary  focus  is  verifying 
practice  implementation.  This  is  accomplished  by  (1)  obtaining  objective  evidence  relevant  to 
the  implementation  of  a  practice,  (2)  comparing  the  objective  evidence  with  what  is  expected, 
and  (3)  making  a  determination  of  practice  implementation  based  on  the  difference  between 
actual  and  expected  evidence. 

The  Plls  assist  the  appraisal  team  (as  well  as  the  implementing  organization)  with  task  1  by 
providing  a  framework  or  structure  that  makes  explicit  the  types  of  objective  evidence  that 
should  be  considered.  In  concert  with  the  People  CMM  model  documentation,  this  provides 
the  model  basis  against  which  the  organization’s  actual  operating  practices  are  compared. 

Note  that  Plls  do  not  prescribe  what  objective  evidence  must  be  present  for  practice 
implementation  determinations  to  be  made.  Instead,  they  make  explicit  what  is  reasonable  for 
an  appraisal  team  to  consider.  The  particular  circumstances  and  attributes  of  the 
organizational  unit  and/or  project  must  be  taken  into  consideration  when  making 
determinations  of  practice  implementation.  As  a  general  rule,  the  more  objective  evidence 
and  the  more  Plls  represented  by  that  objective  evidence,  the  higher  the  confidence  level  that 
the  practice  is  implemented. 

The  PII  structure  helps  the  appraisal  team  compare  the  objective  evidence  with  what  is 
expected,  to  the  extent  that  the  team  has  agreed  in  advance  on  the  objective  evidence  it 
expects  to  see  for  each  process  instantiation  examined.  In  some  cases  it  might  be  difficult  or 
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impossible  to  have  developed  a  team  consensus  in  advance  about  what  objective  evidence 
must  be  seen.  But  sooner  or  later  the  appraisal  team  must  establish  a  consensus  view  of  what 
is  reasonable  since  it  is  only  that  consensus  that  permits  a  determination  of  practice 
implementation. 

The  final  practice  implementation  determination  task  is  developing  a  team  consensus  about 
whether  the  practice  is  implemented  for  the  process  instantiation  being  examined.  This 
decision  is  based  on  the  difference  between  what  is  expected  and  what  is  observed. 
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Appendix  H  Focused  Investigation 
Elaboration  and  Guidance 


Concept  Description 

This  appendix  describes  the  use  of  preliminary  objective  evidence  review,  continuous 
consolidation  of  objective  evidence,  and  practice  characterization  in  focusing  the  data 
collection  and  investigation  effort  of  the  appraisal  team. 

Focused  investigation  relies  on  a  high  degree  of  planning,  organization,  and  subsequent 
management  and  control  of  the  activities  of  the  appraisal.  The  concept  incorporates  the 
following  activities: 

•  collecting  preliminary  objective  evidence  through  instruments  as  part  of  obtaining 
preliminary  data  (e.g.,  with  the  People  CMM  survey) 

•  creating  an  inventory  of  objective  evidence  collected  to  support  the  characterization  of 
practice  implementation 

•  initially  reviewing  and  analyzing  preliminary  objective  evidence  inventoried  to  identify 
gaps  in  objective  evidence  supporting  practice  characterization 

•  identifying  information  needs  for  the  support  of  the  initial  preparation  and  the  refinement 
of  the  data  collection  plan 

•  continuously  consolidating  objective  evidence  collected  and  updating  the  status  of 
practice  characterization  for  each  organizational  unit  instantiation  (aggregated  up  to  the 
organizational  unit) 

Preliminary  Focused  Investigation 

Focused  investigation  should  begin  during  the  appraisal  planning  phase.  Investigation  is  best 
initiated  with  a  practice-based  data  collection  instrument  that  documents  the  organizational 
unit’s  implementation  of  the  practices  of  the  People  CMM  for  each  instantiation  w  ithin  the 
scope  of  the  appraisal.  Preliminary  data  can  be  collected  using  instruments  such  as 
questionnaires,  surveys,  and  presentations.  This  data  should  include  a  preliminary  inventory 
of  practice  implementation  indicators. 

An  inventory  and  review  of  this  data  provides  an  important  initial  determination  of  the  gaps 
in  the  data  available  to  support  practice  implementation,  and  what  information  and  objective 
evidence  is  needed.  These  activities  are  performed  as  part  of  process  2.1.4  Obtain  and 
Analyze  Initial  Objective  Evidence.  The  more  complete  and  comprehensive  this  early  data 
collection  and  analysis,  the  better  prepared  the  organizational  unit  will  be  for  the  appraisal. 
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Preliminary  information  can  provide  the  foundation  for  the  data  collection  plan  for  the 
remainder  of  the  appraisal  process.  It  can  also  provide  the  foundation  for  the  readiness  review 
and  any  necessary  adjustments  in  the  appraisal  plan,  providing  a  clearer  set  of  initial 
expectations  for  the  magnitude  of  the  appraisal  effort.  Data  gaps  found  can  result  in 
additional  document  requests  and  other  data  collection  plans.  These  activities  are  performed 
as  an  early  part  of  process  2.1.5  Prepare  for  Collection  of  Objective  Evidence. 

Continuous  Consolidation  and  Tracking 

A  data  collection  plan  is  developed  and  followed  after  the  initial  focused  investigation  effort. 
Data  collection  activities  are  described  by  process  2.2.1  Examine  Objective  Evidence.  This 
process  typically  consists  of  planned  data  collection  activities  that  include  presentations, 
document  reviews,  and  interviews. 

As  these  data  collection  activities  are  performed,  practice  characterization  and  strengths  and 
weaknesses  are  recorded  and  continuously  reviewed  (see  processes  2.2.2  Verify  and  Validate 
Objective  Evidence  and  2.2.3  Document  Objective  Evidence).  Additional  data  collected  is 
added  to  and  consolidated  with  the  data  already  collected  to  provide  the  Appraisal  team  with 
a  continuous  view  of  their  progress  against  the  data  collection  plan  and  model  coverage.  This 
is  referred  to  as  “continuous  consolidation.” 

Monitoring  and  controlling  the  data  collection  plan  and  model  coverage  is  essential  for 
focused  investigation  and  continuous  consolidation.  The  appraisal  team  must  be  able  to 
record,  monitor,  and  track  progress  against  the  data  collection  plan.  This  can  be  done  in 
several  ways  but  generally  requires  the  use  of  a  mechanism  to  record  progress  toward 
determining  the  practice  characterization  for  each  reference  model  practice  within  the  scope 
of  the  appraisal.  As  data  is  collected  for  each  practice  and  each  sample  instantiation  of  the 
organizational  unit  being  appraised,  some  mechanism  should  be  used  to  make  easier  to 
implement  the  comparison  and  consolidation  practice. 

Perhaps  the  most  important  feature  of  focused  investigation  is  the  appraisal  team’s  awareness 
of  its  status  regarding  the  determination  of  practice  characterization  and  goal  satisfaction.  The 
team  continually  maintains  an  understanding  of  how  the  data  collected  supports  the 
implementation  of  each  practice  for  each  instantiation  of  the  organizational  unit  and  what 
additional  objective  evidence  is  needed.  This  allows  the  team  to  update  the  data  collection 
plan  to  refocus  their  efforts  optimally  during  the  data  collection  activities. 
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Appendix  I 
Guidance 


Combined  Appraisal 


In  determining  appraisal  constraints  (See  activity  2. 1.1.1  Determine  Appraisal  Objectives), 
the  Lead  Appraiser  might  need  to  plan  and  coordinate  for  integration  of  the  SCAMPI  with 
People  CMM  with  other  SCAMPI  appraisals  (e.g.,  SCAMPI  for  CMMI),  as  appropriate  to 
meet  the  organization’s  business  and  appraisal  objectives. 

A  SCAMPI  for  People  CMM  appraisal  can  be  combined  with  a  SCAMPI  for  CMMI 
appraisal,  although  the  considerations  below  should  be  addressed  in  planning  and  preparing 
for  the  combined  appraisal. 

Selecting  a  Lead  Appraiser 

When  the  appraisal  team  leader  performs  a  combined  SCAMPI  with  CMMI  and  People 
CMM,  all  requirements  for  Lead  Appraisers  as  defined  by  the  SEI  for  both  People  CMM  and 
CMMI  must  be  fulfilled.  Refer  to  the  Standard  CMMI  Appraisal  Method  for  Process 
Improvement  ( SCAMPI )  Version  1.1.  Method  Definition  Document  for  details  [SEI  Ola]. 

Lead  appraisers  who  will  conduct  a  combined  SCAMPI  with  CMMI  and  People  CMM 
appraisal  must  be  authorized  lead  appraisers  for  both  models. 

Identifying  Scope 

When  an  organization  wants  a  combined  appraisal  for  multiple  models,  this  becomes  the 
issue  for  the  Lead  Appraiser.  It  is  then  the  responsibility  of  the  Lead  Appraiser  to  integrate 
the  activities  involved  in  appraising  the  organization  against  each  model.  It  is  also  the 
responsibility  of  the  Lead  Appraiser  to  ensure  that  the  organization  is  prepared  for  an 
integrated  appraisal  and  that  the  appropriate  organizational  scope  and  participants  are 
identified. 

Planning  the  Appraisal 

When  a  combined  SCAMPI  with  CMMI  and  People  CMM  appraisal  is  planned  there  should 
be  one  combined  and  common  appraisal  plan. 
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Team  Preparation 

For  a  SCAMPI  with  People  CMM,  all  team  members  must  have  previously  completed  the 
SEI-licensed  Introduction  to  People  CMM  course  delivered  by  an  instructor  who  is 
authorized  by  the  SEI. 

When  a  combined  SCAMPI  with  CMMI  and  People  CMM  is  performed,  ideally  all  team 
members  will  have  completed  both  the  SEI-licensed  Introduction  to  CMMI  course  and  the 
Introduction  to  People  CMM  course  delivered  by  instructors  who  are  authorized  by  the  SEI. 
When  a  combined  SCAMPI  with  CMMI  and  People  CMM  is  performed,  some  appraisal 
team  members  might  not  be  trained  to  participate  with  both  models.  That  is,  some  might  only 
be  trained  for  CMMI  and  some  for  People  CMM.  In  these  cases,  the  appraisal  team  members 
who  have  only  one  model  capability  can  sit  with  the  team  throughout  the  consensus  but 
cannot  participate  in  the  consensus  discussions  for  the  models  in  which  they  are  not  trained 
and  qualified. 
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Appendix  J  Process  Area  Workbook 

(PAWB) 


The  process  area  workbook  (PAWB)  is  a  spread  sheet  tool  for  capturing  evidence  during  an 
appraisal.  Evidence  gathered  includes  direct  artifacts,  indirect  artifacts,  interviews,  and 
survey  data  (if  surveys  are  used).  While  use  of  the  PAWB  is  not  a  SCAMPI  requirement,  it  is 
extremely  useful.  Figure  2  is  an  example  worksheet  for  a  work  environment. 


Figure  2:  Work  Environment  Worksheet  in  PAWB 


CMU/SEI-2005-SR-001 


139 


Figure  2  shows,  by  project,  the  type  of  evidence  obtained  for  each  goal  and  for  each  practice  within 
the  goal  in  the  appraisal  scope.  The  appraisal  team  must  keep  in  mind  that  more  than  one  piece  of 
evidence  is  required  before  the  instantiation  characterization  can  be  determined  in  column  5. 

The  PAWB  worksheets  are  evolving  documents  used  by  the  appraisal  team  to  understand  the 
degree  of  evidence  capture  and  work  required  before  the  appraisal  can  be  completed.  The 
worksheets  should  be  reviewed  daily.  Definitions  and  requirements  for  each  column  in  the 
PAWB  are  listed  below. 

1.  Goal— The  goal  column  has  a  row  for  each  specific  goal  and  each  generic  goal  for  a 
process  area  (rows  G1  through  Gn,  including  the  institionalization  goal.) 

2.  Practice — The  practice  column  maps  practices  to  goals,  with  a  row  for  each  practice 
associated  with  the  goal. 

3.  Project — For  each  practice,  the  column  has  a  row  for  each  project  included  in  the 
appraisal  for  the  appraised  organization.  If  more  than  four  projects  are  included,  insert  a 
row  for  each  additional  project  by  placing  the  cursor  in  the  project  four  cell,  right 
clicking  your  mouse,  and  selecting  “insert”  and  then  “entire  row.” 

Table  8:  Indicator  Types 


Indicator  Type 

Description 

Examples 

Direct  artifacts 

The  tangible  outputs  resulting  directly 
from  implementation  of  a  practice.  An 
integral  part  of  verifying  practice 
implementation.  Might  be  explicitly 
stated  or  implied  by  the  practice 
statement  or  associated  informative 
material. 

Documents,  deliverable 
work  products,  training 
materials,  plans,  etc. 

Work  products  are 
sometimes  stated  in  the 
informative  materials  in 
the  People  CMM 

Indirect  artifacts 

Artifacts  that  are  a  consequence  of 
performing  a  practice  or  that  substantiate 
its  implementation,  but  which  are  not  the 
purpose  for  which  the  practice  is 
performed.  This  indicator  type  is 
especially  useful  when  there  might  be 
doubts  about  whether  the  intent  of  the 
practice  has  been  met  (e.g.,  an  artifact 
exists  but  there  is  no  indication  of  where 
it  came  from,  who  worked  to  develop  it, 
or  how  it  is  used. 

Meeting  minutes,  review 
results,  status  reports 

Performance  measures 

Work  products  are 
sometimes  stated  in  the 
informative  materials  in 
the  People  CMM. 

Affirmations 

Oral  or  written  statements  confirming 
or  supporting  implementation  of  a 
practice.  These  are  usually  provided  by 
the  implementer  of  the  practice  and/or 
internal  or  external  customers,  but 
might  also  include  other  stakeholders 
(e.g.,  managers,  suppliers). 

Interviews 

Presentations 

Questionnaire  responses 

140 


CMU/SEI-2005-SR-001 


4.  Type — Mini-teams  use  this  column  to  specify  the  type  of  objective  evidence  found  as 
practice  implementation  is  characterized  at  the  instantiation  (project)  level.  Possible 
types  include  the  following:  “A”  for  direct  artifacts,  “B”  for  indirect  artifacts,  and  “C” 
for  affirmations.  For  type  C,  face-to-face  (F2F)  should  also  be  recorded  if  the  objective 
evidence  was  collected  from  discussion  with  practitioners  so  the  appraisal  requirement 
for  type  C  coverage  rule  can  be  tracked. 

5.  Instantiation  Characterization — Mini-teams  use  this  column  to  characterize  the 
degree  of  practice  implementation  at  the  instantiation  (project)  level.  Practice 
implementation  is  characterized  as  fully  implemented  (FI),  largely  implemented  (LI), 
partially  implemented  (PI),  or  not  implemented  (NI). 


Table  9:  Rules  for  Characterizing  Instantiation-Level  Implementations  of 
Practices 


Label 

Meaning 

Fully  Implement  (FI) 

•  The  direct  artifact  is  present  and  judged  to  be  appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to  ^ 

confirm  the  implementation. 

•  No  substantial  weaknesses  were  noted. 

Largely  Implemented  (LI) 

•  The  direct  artifact  is  present  and  judged  to  be  appropriate. 

•  At  least  one  direct  artifact  and/or  affirmation  exists  confirm 
the  implementation. 

•  One  or  more  weaknesses  were  noted. 

Partially 

Implemented  (PI) 

•  The  direct  artifact  is  absent  or  judged  to  be  inadequate. 

•  Artifacts  or  affirmations  suggest  that  some  aspects  of  the 
practice  are  implemented. 

•  Weaknesses  have  been  documented. 

Not  Implemented  (NI) 

Any  situation  not  covered  above. 

6.  Weakness — Mini-teams  use  this  column  to  document  gaps  in  the  practice 
implementation  at  the  instantiation  (project)  level.  Weaknesses  must  be  recorded  for  LI 
and  PI  characterizations. 

7.  Information  Needed — Mini-teams  use  this  field  to  record  any  additional  information 
needed  from  the  organization  in  order  to  finalize  the  instantiation  characterization. 

8.  Organizational/Unit  (OU)  Characterization — The  full  appraisal  team  uses  this  field  to 
record  the  practice  implementation  aggregation.  The  same  labels  described  above  are 
used,  but  the  characterization  is  determined  for  the  organizational  level. 
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Table  10:  Rules  for  Aggregating  Instantiation-Level  Characterizations  to  Derive 
Appraised  Organization-Level  Characterizations 


Condition 

Outcome 

Remarks 

All  X  (e.g.,  all  LI) 

X 

All  instantiations  have  the  same  characterization. 

All  LI  or  FI 

LI 

All  instantiations  are  characterized  LI  or  higher. 

Any  PI,  No  NI 

LI  or  PI 

Team  judgment  is  applied  to  choose  LI  or  PI  for  the 
organizational  unit. 

Any  NI 

NI,  PI,  or 
LI 

Team  judgment  is  applied  to  choose  LI  or  PI  for  the 
organizational  unit. 

9.  OU  Preliminary  Finding— The  full  appraisal  team  uses  this  column  to  document  gaps 
found  with  the  practice  implementation  at  the  organizational  level.  Weaknesses  must  be 
recorded  for  LI,  PI,  and  NI  characterizations. 

10.  Goal  Rating— The  full  appraisal  team  uses  this  column  to  rate  goal  satisfaction  based 
upon  the  extent  of  practice  implementation.  The  goal  is  rated  satisfied  if 

•  all  associated  practices  are  characterized  at  the  organizational  unit  level  as  either 
LI  or  FI 

•  the  aggregation  of  weaknesses  associated  with  the  goal  does  not  have  a 
significant  negative  impact  on  goal  achievement 

11.  Goal-Level  Findings— The  full  appraisal  team  uses  this  column  to  document  goal-level 
statements  that  summarize  the  gaps  in  practice  implementation  (strengths  are  optional). 
Normally,  these  statements  must  be  abstracted  to  the  level  of  the  organization. 

12.  Supporting  Evidence— The  full  appraisal  team  uses  this  column  to  summarize  the 
evidence  that  supports  the  goal-level  findings. 
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Appendix  K  Use  of  People  CMM  Survey 


A  survey  instrument  was  required  for  all  People  CMM  assessments  as  described  in  the  People 
CMM  BAM.  Rules  for  selecting  the  number  of  participants  and  ensuring  demographic 
coverage  for  the  surveys  are  defined  in  the  People  CMM-Based  Assessment  Method 
Description  [Hefley  98] .  Even  prior  to  SCAMPI,  there  were  indications  that  the  survey  was 
not  an  efficient  or  effective  approach  to  understanding  areas  of  concerns  for  higher  maturity 
organizations  using  the  People  CMM. 

The  pilot  SCAMPI  with  People  CMM  with  the  TCS  enterprise  appraisal  was  granted  a 
waiver  on  the  use  of  the  survey.  CMMI  SCAMPI  does  not  require  a  survey;  however, 
instruments  such  as  surveys  or  questionnaires  may  be  used  as  identified  in  the  data 
collection  plan  [SEI  Ola].  The  SEI  recommends  that  the  People  CMM  appraisal  survey  be 
considered  for  use  in  low  maturity  level  organizations  when  performing  a  SCAMPI  with 
People  CMM  appraisal. 
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Appendix  L  Terminology 


There  are  some  key  terminology  differences  between  the  CMMI  and  People  CMM  model 
that  should  be  kept  in  mind  when  applying  SCAMPI  to  both. 

In  CMMI: 

1.  Results  from  projects  determine  the  level  of  implementation  of  the  practices. 

2.  PLs  are  interviewed  representing  selected  projects. 

3.  FARs  are  also  interviewed  from  the  projects. 

4.  All  selected  projects  are  aggregated  into  the  organization  report. 

In  People  CMM: 

1.  The  equivalent  to  “projects”  results  from  the  organization,  and  selected  units  determine  ^ 
the  level  of  implementation  of  practices 

2.  Organizational  artifacts  might  propagate  to  units  as  direct  or  indirect  artifact  evidence. 

3.  Affirmations  from  the  organization  representatives  are  taken  in  addition  to  unit  (project) 
representatives. 

4.  Affirmations  from  PLs  and  ICs  (~FARs)  are  taken. 

5.  The  organization  and  PL/ICs  are  aggregated  into  the  organization  report. 

The  People  CMM-Based  Assessment  Method  V  1.0  uses  the  term  “assessment”  for  the 
method  to  evaluate  the  maturity  of  an  organization  when  using  the  People  CMM  as  a 
reference.  In  the  interpretation  guide  in  Section  2,  the  term  “appraisal”  is  used  to  address  the 
People  CMM  “assessment”  as  consistent  with  CMMI  SCAMPI.  Lead  Assessor  becomes 
Lead  Appraiser. 

Where  CMMI  SCAMPI  uses  “appraised  organization,”  SCAMPI  with  People  CMM  uses 
“organization.”  Where  CMMI  SCAMPI  uses  “projects,”  SCAMPI  with  People  CMM  uses 
“workgroups”  and  “units.”  “Middle  managers”  are  all  managers  above  the  unit  leader  and 
below  the  senior  manager.  “Engineers”  in  CMMI  SCAMPI  are  “managers  and  individuals"  in 
SCAMPI  with  People  CMM.  “Practitioners)”  in  CMMI  become  “managed  sf  or 
“individual(s)”  or  “individual  contributors”  in  People  CMM.  “Functional  area  representatives 
(FARs),”  who  are  the  non-managers  in  CMMI  SCAMPI,  become  “individuals”  in  People 
CMM  SCAMPI,  which  includes  the  process  owners. 

“Unit”  in  People  CMM  is  a  recursive  definition  for  any  organizational  entity  starting  with 
“workgroup.”  It  can  mean  any  substructure  within  an  organization  (e.g.,  project,  division. 
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sector).  “Unit”  is  typically  the  lowest  level  in  the  organization  where  maturity  level  2 
workforce  activities  (such  as  staffing  and  performance  management)  happen. 
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Appendix  M  Mixed  Goals 


Mixed  goals  are  those  which  have  practices  at  both  the  unit  level  and  the  organizational  level. 
An  example  in  People  CMM  would  be  empowered  work  groups  (EWG)  goal  3: 

P8:  Responsibility  and  authority  for  performing  selected  workforce  activities  is  delegated  to 
empowered  workgroups. 

P9:  Empowered  workgroups  tailor  workforce  activities  delegated  to  them  and  plan  for  their 
adoption. 

P10:  Empowered  workgroups  perform  the  workforce  activities  delegated  to  them. 

Pll:  Empowered  workgroups  participate  in  managing  their  performance. 

P8  can  only  be  performed  by  an  entity  above  the  workgroup  (unit)  and  would  thus  be  at  an 
“organization”  level.  P9,  P10,  and  Pll  all  can  be  performed  by  the  workgroup  (unit). 

Mixed  goals  become  important  when  the  appraisal  team  needs  to  demonstrate  sufficient 
coverage  and  evidence  across  the  organization  for  artifacts  and  interviews.  The  appropriate 
level  in  the  organization  must  provide  the  artifacts  and  participate  in  the  interviews  for  the 
appraisal  team  to  get  an  understanding  of  compliance  or  the  lack  of  compliance  with  the  practice. 

Mixed  goals  become  especially  important  when  the  50%  rule  is  being  reviewed.  Because  the 
50%  rule  was  written  for  units  in  SCAMPI,  it  needs  to  be  reinterpreted  when  goals  are  mixed. 
In  the  example  of  goal  3  for  EWG  the  interview  evidence  can  come  from  units  for  P9,  P10, 
and  Pll,  but  for  P8  can  only  come  from  an  organizational  entity. 

For  example,  where  there  are  4  units  in  the  appraisal,  the  matrix  must  include  the 
organization  as  an  entity  along  with  the  4  units,  as  in  Table  11. 


Table  11:  Mixed  Goals  Matrix 


Organization 
Entity  .  £ 

Unit  1 

Unit  2 

Unit  3 

Unit  3 

I  Row 

P8 

P9 

P10 

Pll 

£  Column 
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Because  only  the  organization  entity  can  provide  interview  evidence  for  P8,  it  becomes 
especially  critical  that  the  appraisal  team  understand  this  and  address  it  in  plans  and  scripts. 
Furthermore,  if  the  interview  evidence  is  not  provided  by  the  organizational  entity,  the  goal 
cannot  be  satisfied,  as  the  £  row  cannot  be  satisfied.  This  rightly  places  increased  emphasis 
on  ensuring  that  interviews  are  structured  to  incorporate  this  understanding  to  ask  the 
question  for  P8,  in  this  example.  Each  institutionalization  goal  in  each  PA  is  mixed  in  People 
CMM. 

Other  goals  which  are  mixed  include  the  following: 

•  STF  G1  (0=6;  U=7  and  8) 

•  STF  G2  (0=7  and  9;  U=6  and  8) 

•  STF  G4  (0=  1 ;  U=2, 4,  and  5) 

•  PM  G4  (0=14;  U=13) 

•  WFP  G3  (0=1 1 ;  U=6  and  7) 

•  WGD  G2  (0=4;  U=7,  8, 10, 1 1,  and  12) 

•  Cl  G3  (0=  5,  8,  and  9;  U=7) 

•  EWG  G1  (0=3, 4,  and  5;  U=1  and  2) 

•  MTR  G1  (0=10;  U=l,  2,  3,  and  4 

•  OPA  G1  (0=3;  U=1  and  2) 

During  the  appraisal  it  must  be  determined  if  any  goal  is  designed  to  cover  both 
organizational  support  for  and  lower  organizational  level  (unit  or  individual)  implementation 
of  a  set  of  workforce  practices.  If  so,  the  appraisal  team  must  obtain  information  from  the 
appropriate  level  in  the  appraised  organization.  These  goals  might  thus  become  mixed  in  the 
appraised  organization  if  the  organization  chose  to  implement  the  practices  at  multiple 
organizational  levels.  An  example  is  STF  G2,  which  could  be  implemented  at  an  organization 
or  unit  level.  Another  example  is  Cl  Gl:  PI  is  organization  level,  P2,  3, 12  practices  are 
dependent  on  implementation. 

Practices  can  be  covered  at  multiple  levels  depending  on  how  the  organization  is  structured 
For  example,  PC  G3,  Cl  Gl,  CBA  Gl,  CBA  G3,  and  QPM  Gl. 

Mixed  goals  also  occur  in  CMMI.  For  example, 

•  OEI  SGI  (OSP1.1-1, ;  U=SPl-2,  SP1-3) 

•  each  institutionalization  goal  in  each  PA 

-  generic  practices  at  the  organization  level  are  GP  2.1, 3.1,  and  3.2 

—  generic  practices  at  the  project  level  are  GP  2.2, 2.3, 2.4,  2.5,  2.6,  2.7, 2.8.  2.9,  and 

2.10 
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Glossary 


This  glossary  merges  terms  from  several  different  sources.  Original  sources  are  noted  after 

An  observation  extracted  from  data  collected  during  an  appraisal 
that  has  been  determined  by  the  appraisal  team  to  be  (a)  worded 
appropriately,  (b)  based  on  information  seen  or  heard,  (c)  relevant 
to  the  appraisal  reference  model  being  used,  (d)  significant  such 
that  it  can  be  classified  as  a  strength,  weakness,  or  alternative 
practice,  and  (e)  not  redundant  with  other  observations. 

(from  ARC  VI. 1  [SEI  01b]) 

An  oral  of  written  statement  confirming  or  supporting 
implementation  of  a  People  CMM  model  specific  practice  or 
generic  practice.  Affirmations  are  usually  provided  by  the 
implementers  of  the  practice  and/or  internal  or  external  customers, 
but  might  also  include  other  stakeholders  (e.g.,  managers, 
suppliers). 

(derived  from  MDD  method  overview  [SEI  01a]) 

Interview  responses  are  examples  of  face-to-face  affirmations. 
Alternative  forms  of  affirmations  could  include  presentations  or 
demonstrations  of  a  tool  or  mechanism  as  it  relates  to 
implementation  of  a  People  CMM  model  practice. 

(derived  from  MDD  Pll  Appendix  B  [SEI  01a]) 

alternative  practice  A  practice  that  is  a  substitute  for  one  or  more  generic  or  specific 

practices  contained  in  the  People  CMM  model  that  achieves  an 
equivalent  effect  toward  satisfying  the  goal  associated  with  the 
practices.  Alternative  practices  are  not  necessarily  one-for-one 
replacements  for  the  generic  or  specific  practices. 

(from  ARC  VI. 1  [SEI  01b],  CMMI  model  glossary  [SEI  02],  and  the  People  CMM 
BAM  Description  [Hefley  98]) 


the  definitions. 

accurate 

observation 


affirmation 
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appraisal 


appraisal 

constraints 


appraisal 
disclosure 
statement  (ADS) 


appraisal  findings 


appraisal  input 


appraisal  method 
class 


appraisal  modes  of 
usage 


An  examination  of  one  or  more  processes  by  a  trained  team  of 
professionals  using  an  appraisal  reference  model  as  the  basis  for 
determining,  as  a  minimum,  strengths  and  weaknesses. 

(from  ARC  VI.  1  [SEI  01b]) 

An  appraisal  that  an  organization  does  to  and  for  itself  for  the 
purposes  of  process  improvement. 

(from  ARC  V1.1  [SEI  01b]) 

Constraints  that  affect  assessment  conduct,  such  as  budget 
limitations,  schedule  limitations,  and  resource  limitations  (people 
and  facilities). 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  summary  statement  describing  the  ratings  generated  as  outputs  of 
the  appraisal,  and  the  conditions  and  constraints  under  which  the 
appraisal  was  performed.  The  ADS  should  be  used  for  public 
disclosures  of  maturity  level  or  capability  level  ratings  so  they  can 
be  interpreted  accurately. 

(from  this  document) 

The  results  of  an  appraisal  that  identify  the  most  important  issues, 
problems,  or  opportunities  for  process  improvement  within  the 
appraisal  scope.  Appraisal  findings  are  inferences  drawn  from  valid 
observations. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI. 1  [SEI  01b]) 

The  collection  of  appraisal  information  required  before  data 
collection  can  commence. 

(from  ISO  98C  [ISO  98]  and  ARC  VI  .1  [SEI  01b]) 

A  family  of  appraisal  methods  that  satisfy  a  defined  subset  of 
requirements  in  the  Appraisal  Requirements  for  People  CMM 
(ARC).  These  classes  are  defined  so  as  to  align  with  typical  usage 
modes  of  appraisal  methods. 

(derived  from  ARC  VI. 0  [SEI  00],  CMMI  model  glossary  [SEI  02],  and  ARC  VI. 1 
[SEI  01b]) 

The  contexts  in  which  an  appraisal  method  might  be  utilized.  Appraisal 
modes  of  usage  identified  for  the  SCAMPI  method  include  internal 
process  improvement,  supplier  selection,  and  process  monitoring. 
Appraisal  modes  of  usage  identified  for  the  SCAMPI  with  People 
CMM  method  are  limited  to  internal  process  improvement. 
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appraisal 

objectives 

appraisal  output 

appraisal 

participants 

appraisal  rating 


appraisal  record 

appraisal  reference 
model 

appraisal 

requirements 

appraisal  scope 

appraisal  sponsor 


The  desired  outcome(s)  of  an  appraisal  process. 

(from  ARC  VI. 1  [SEI  01b]) 

All  of  the  tangible  results  from  an  appraisal.  (See  “appraisal 
record.”) 

(from  ISO  98C  [ISO  98]  and  ARC  VI. 1  [SEI  01b]) 


Members  of  the  organizational  unit  who  participate  in  providing 
information  during  the  appraisal. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI. 1  [SEI  01b]) 

The  value  assigned  by  an  appraisal  team  to  either  (a)  a  People 
CMM  goal  or  process  area,  (b)  the  capability  level  of  a  process 
area,  or  (c)  the  maturity  level  of  an  organizational  unit.  The  rating  is 
determined  by  enacting  the  defined  rating  process  for  the  appraisal 
method  being  employed. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI. 1  [SEI  01b]) 

An  orderly,  documented  collection  of  information  that  is  pertinent 
to  the  appraisal  and  adds  to  the  understanding  and  verification  of 
the  appraisal  findings  and  ratings  generated. 

(derived  from  ISO  98C  [ISO  98]  and  ARC  VI  .1  [SEI  01  b]) 

The  People  CMM  model  to  which  an  appraisal  team  correlates 
implemented  process  activities. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI. 1  [SEI  01b]) 

Appraisal  constraints  and  goals. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  definition  of  the  boundaries  of  the  appraisal  encompassing  the 
organizational  limits  and  the  People  CMM  model  limits  within 
which  the  processes  to  be  investigated  operate. 

(derived  from  CMMI  model  glossary  [SEI  02],  ISO  98C  [ISO  98],  and  ARC  VI. 1 
[SEI  01b]) 

The  individual,  internal  or  external  to  the  organization  being 
appraised,  who  requires  the  appraisal  to  be  performed  and  provides 
financial  or  other  resources  to  carry  it  out. 

(derived  from  ISO  98C  [ISO  98]  and  ARC  VI  .1  [SEI  01b]) 


CMU/SEI-2005-SR-001 


151 


appraisal  tailoring 


appraisal  team 
leader 


appraised  entity 


artifact 


coach 


confidentiality 


consensus 


Selection  of  options  within  the  appraisal  method  for  use  in  a 
specific  instance.  The  intent  of  tailoring  is  to  assist  an  organization 
in  aligning  application  of  the  method  with  its  business  needs  and 
objectives. 

(from  CMMI  model  glossary  [SEI 02]  and  ARC  VI. 1  [SEI  01b]) 

The  person  who  leads  the  activities  of  an  appraisal  and  has  satisfied 
the  qualification  criteria  for  experience,  knowledge,  and  skills 
defined  by  the  appraisal  method. 

(from  ARC  VI  .1  [SEI  01b]) 

The  organizational  units  to  which  appraisal  outputs  apply.  An 
appraised  entity  might  be  any  portion  of  an  organization  including 
an  entire  company,  a  selected  business  unit,  a  specific  geographic 
site,  units  supporting  a  particular  product  line,  units  involved  in  a 
particular  type  of  service,  an  individual  project,  or  a  multi-company 
team. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  tangible  form  of  objective  evidence  indicative  of  work  being 
performed  that  is  a  direct  or  indirect  result  of  implementing  a 
People  CMM  model  practice.  (See  “direct  artifact”  and 
“indirect  artifact.”) 

An  authorized  lead  appraiser  who  shares  the  responsibilities  of  the 
appraisal  team  leader  with  another  qualified  person.  It  must  be 
made  clear  to  the  appraisal  team  members  and  to  the  sponsor  which 
responsibilities  are  being  shared  and  by  whom.  An  authorized  lead 
appraiser  has  the  ultimate  responsibility  for  the  appraisal  activities 
and  the  results,  whether  acting  in  the  role  of  appraisal  team  leader 
or  coach. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

An  agreement  by  which  data  will  not  be  attributed  to  a  particular 
individual,  unit,  or  organization,  or  be  disclosed  without  prior 
agreement  or  authorization. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  method  of  decision  making  that  allows  team  members  to  develop 
a  common  basis  of  understanding  and  develop  general  agreement 
concerning  a  decision  that  all  team  members  are  willing  to  support, 
(from  ARC  V1.1  [SEI  01b]) 
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consistency 


consolidation 


corroboration 


coverage 


coverage  criteria 


data  collection 
session 


The  degree  of  uniformity,  standardization,  and  freedom  from 
contradiction  among  documents  or  system  components. 

Consistency  of  an  appraisal  method  refers  to  the  ability  of  different 
appraisal  teams  using  the  same  method  to  conduct  appraisals  of  the 
same  scope  to  produce  non-conflicting  results. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  activity  of  collecting  and  summarizing  the  information 
provided  into  a  manageable  set  of  data  to  (a)  determine  the  extent  to 
which  the  data  are  corroborated  and  cover  the  areas  being 
investigated,  (b)  determine  the  data’s  sufficiency  for  making 
judgments,  and  (c)  revise  the  data-gathering  plan  as  necessary  to 
achieve  this  sufficiency. 

(from  ARC  VI. 1  [SEI  01b]) 

The  extent  to  which  enough  data  has  been  gathered  to  confirm  that 
an  observation  is  acceptable  for  use  by  an  appraisal  team. 

(from  ARC  V1.1  [SEI  01b]) 

In  SCAMPI,  corroboration  is  obtained  through  method 
requirements  for  the  collection  of  practice  implementation 
indicators  of  multiple  types.  (See  “practice  implementation 
indicator.”) 

The  extent  to  which  objective  evidence  gathered  addresses  a  model 
component  within  the  scope  of  an  appraisal. 

(from  ARC  VI. 1  [SEI  01b]) 

The  specific  criterion  that  must  be  satisfied  in  order  for  coverage  to 
be  claimed. 

(from  ARC  VI  .1  [SEI  01b]) 

An  activity  during  which  information  that  will  later  be  used  as  the 
basis  for  observation  formulation  or  corroboration  is  gathered.  Data 
collection  sessions  (or  activities)  include  the  administration  and/or 
analysis  of  instruments,  document  review,  interviews,  and 
presentations. 

(from  ARC  VI.  1  [SEI  01b]) 
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direct  artifact 


discovery-based 

appraisal 


document 


effective  process 

FI 

finding 


findings 


The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice 
implementation.  Might  be  explicitly  stated  or  implied  by  the 
practice  statement  or  associated  informative  material. 

(MDD  method  overview  [SEI  01a]) 

An  appraisal  in  which  limited  objective  evidence  is  provided  by  the 
appraised  organization  prior  to  the  appraisal,  and  the  appraisal  team 
must  probe  and  uncover  a  majority  of  the  objective  evidence 
necessary  to  obtain  sufficient  coverage  of  People  CMM  model 
practices.  Discovery-based  appraisals  typically  involve  substantially 
greater  appraisal  team  effort  than  verification-based  appraisals,  in 
which  much  of  the  objective  evidence  is  provided  by  the  appraised 
organization.  (See  “verification-based  appraisal”  for  contrast.) 

(from  MDD  method  overview  [SEI  01a]) 

A  collection  of  data,  regardless  of  the  medium  on  which  it  is 
recorded,  that  generally  has  permanence  and  can  be  read  by  humans 
or  machines. 

(from  ARC  V1.1  [SEI  01b]) 

In  SCAMPI,  documents  are  work  products  reflecting  the 
implementation  of  one  or  more  model  practices.  This  typically 
includes  work  products  such  as  organizational  policies,  procedures, 
and  implementation-level  work  products.  Documents  might  be 
available  in  hardcopy,  softcopy,  or  accessible  via  hyperlinks  in  a  Web- 
based  environment. 

(derived  from  MDD  method  overview  [SEI  01a]) 

A  process  that  can  be  characterized  as  practiced,  documented, 
enforced,  trained,  measured,  and  able  to  improve. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

(See  “fully  implemented.”) 

An  observation  or  collection  of  observations  that  have  been 
accepted  by  the  team  as  valid.  A  finding  includes  strengths, 
weaknesses,  evidence  of  alternative  practices,  and  evidence  of  non- 
applicable  practices.  A  set  of  findings  should  be  accurate, 
corroborated,  and  consistent  within  itself. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  conclusions  of  an  appraisal,  evaluation,  audit,  or  review  that 
identify  the  most  important  issues,  problems,  or  opportunities 
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focused 

investigation 


fully  implemented 
(FI) 


indirect  artifact 


instantiation 


institutionalization 


instruments 


within  the  appraisal  scope.  Findings  include,  at  a  minimum, 
strengths  and  weaknesses  based  on  valid  observations. 

(from  ARC  VI. 1  [SEI  01b]) 

A  technique  to  prioritize  appraisal  team  effort  based  on  the 
continuous  collection  and  consolidation  of  appraisal  data,  and 
monitoring  of  progress  toward  achieving  sufficient  coverage  of 
People  CMM  model  practices.  Appraisal  resources  are  targeted 
toward  those  areas  for  which  further  investigation  is  needed  to 
collect  additional  data  or  verify  the  collected  set  of  objective 
evidence. 

(derived  from  MDD  method  overview  [SEI  01a]) 

A  practice  characterization  value  assigned  to  a  process  instantiation 
when  (1)  direct  artifacts  are  present  and  judged  to  be  appropriate, 
(2)  at  least  one  indirect  artifact  and/or  affirmation  exists  to  confirm 
the  implementation,  and  (3)  no  substantial  weaknesses  are  noted, 
(from  MDD  3.7.2  [SEI  01a]) 

An  artifact  that  is  a  consequence  of  performing  a  specific  or  generic 
practice  or  that  substantiates  its  implementation,  but  which  is  not 
the  purpose  for  which  the  practice  is  performed.  This  indicator  type 
is  especially  useful  when  there  might  be  doubts  about  whether  the 
intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists  but 
there  is  no  indication  of  where  it  came  from,  who  worked  to 
develop  it,  or  how  it  is  used). 

(from  MDD  method  overview  [SEI  01a]) 

For  practices  implementing  projects,  each  project  can  demonstrate 
instances  of  consistent  performance.  For  practices  implemented 
organization-wide,  instances  of  consistent  performance  are  in 
evidence  organizationally. 

(from  MDD  method  overview  [SEI  01a]) 

The  building  of  infrastructure  and  corporate  culture  that  support 
methods,  practices,  and  procedures  so  that  they  are  the  ongoing 
way  of  doing  business,  even  after  those  who  originally  defined 
them  are  gone. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of 
data  (e.g.,  questionnaires,  organizational  unit  information  packets), 
(from  ARC  VI. 1  [SEI  01b]) 

In  SCAMPI,  instruments  are  used  to  collect  written  information 
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internal  process 
improvement  (IPI) 


interviews 


largely 

implemented  (LI) 


lead  appraiser 

LI 

manager 


relative  to  the  organizational  unit’s  implementation  of  People  CMM 
model  practices.  This  can  include  assets  such  as  questionnaires, 
surveys,  or  an  organizational  mapping  of  People  CMM  model 
practices  to  its  corresponding  processes. 

An  appraisal  mode  of  usage  in  which  organizations  appraise 
internal  processes,  generally  to  either  baseline  their  process 
capability,  to  establish  or  update  a  process  improvement  program, 
or  to  measure  progress  in  implementing  such  a  program. 

(derived  from  MDD  method  overview  [SEI  01a]) 

A  meeting  of  appraisal  team  members  with  appraisal  participants 
for  the  purpose  of  gathering  information  relative  to  work  processes 
in  place. 

(from  ARC  VI. 1  (SEI  01b]) 

In  SCAMPI,  this  includes  face-to-face  interaction  with  those 
implementing  or  using  the  processes  within  the  organizational  unit. 
Interviews  are  typically  held  with  various  groups  or  individuals, 
such  as  project  leaders,  managers,  and  practitioners.  A  combination 
of  formal  and  informal  interviews  might  be  held  and  interview  scripts 
or  exploratory  questions  developed  to  elicit  the  information  needed. 

A  practice  characterization  value  assigned  to  a  process  instantiation 
when  (1)  direct  artifacts  are  present  and  judged  to  be  appropriate, 

(2)  at  least  one  indirect  artifact  and/or  affirmation  exists  to  confirm 
the  implementation,  and  (3)  one  or  more  weaknesses  are  noted. 

(from  MDD  3.7.2  [SEI  01a]) 

A  person  who  has  achieved  recognition  from  an  authorizing  body  to 
perform  as  an  appraisal  team  leader  for  a  particular  appraisal 
method. 

(from  ARC  VI. 1  (SEI  01b]) 

(See  “largely  implemented.”) 

A  role  that  encompasses  providing  technical  and  administrative 
direction  and  control  to  individuals  performing  tasks  or  activities 
within  the  manager’s  area  of  responsibility.  The  traditional 
functions  of  a  manager  include  planning,  allocating  resources, 
organizing,  directing,  and  controlling  work  within  an  area  of 
responsibility. 

(from  People  CMM  BAM  Description  (Hefley  98]) 
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MQ 

NA 

Nl 
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not  applicable  (NA) 


not  implemented 
(Nl) 


not  rated  (NR) 


A  well-defined  evolutionary  plateau  toward  achieving  a  mature 
organizational  process. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  set  of  questionnaires  that  sample  the  key  practices  in  each  key 
process  area  of  the  People  CMM.  The  maturity  questionnaire  is 
used  as  a  springboard  to  gather  data  across  an  organization  and 
appraise  the  capability  of  an  organization  to  execute  its  processes 
reliably. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

In  SCAMPI  with  People  CMM,  the  MQ  is  an  example  of  an 
instrument. 

(from  this  document) 

The  site  representatives  who  fall  between  the  project  leaders  and  the'f 
site  manager  in  the  organizational  hierarchy. 

(from  People  CMM  BAM  Description  [Curtis  01]) 

(See  “process  area  mini-team.”) 

(See  “maturity  questionnaire”) 

(See  “not  applicable.”) 

(See  “not  implemented.”) 

An  observation  that  is  believed  to  have  a  significant  impact  on  the 
organization’s  capability  but  is  not  related  to  a  particular  component 
of  the  People  CMM. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Rating  given  to  a  People  CMM  component  that  is  either  not 
applicable  or  insignificant  in  an  organization’s  business 
environment. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  practice  characterization  value  assigned  when  the  appraisal  team 
determines  insufficient  objective  evidence  exists  to  state  that  the 
practice  is  implemented.  That  is,  the  criteria  for  assigning  a  value  of 
fully  implemented  (FI),  largely  implemented  (LI),  or  partially 
implemented  (PI)  are  not  satisfied. 

(from  this  document) 

Rating  given  to  a  People  CMM  component  that  falls  outside  the 
scope  of  an  appraisal  and  to  People  CMM  components  for  which 
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NR 

objective  evidence 


observation 


organization 


organizational 

scope 


organizational  unit 


the  appraisal  team  did  not  obtain  coverage. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

(See  “not  rated.”) 

Qualitative  or  quantitative  information,  records,  or  statements  of 
fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the 
existence  and  implementation  of  a  process  element,  which  is  based 
on  observation,  measurement,  or  test  and  which  can  be  verified. 

(from  CMMI  model  glossary  [SEI  02],  ISO  98C  [ISO  98]  and  ARC  VI  .1  [SEI 01  b]) 

In  SCAMPI,  sources  of  objective  evidence  include  instruments, 
presentations,  documents,  and  interviews. 

A  written  record  that  represents  the  appraisal  team  members’ 
understanding  of  information  either  seen  or  heard  during  the 
appraisal  data  collection  activities.  The  written  record  might  take 
the  form  of  a  statement  or  might  take  alternative  forms  as  long  as 
the  information  content  is  preserved. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI. 1  [SEI  01b]) 

Information  extracted  from  the  notes  of  data  collection  sessions, 
(from  People  CMM  BAM  Description  [Hefley  98]) 

A  unit  within  a  company  or  other  entity  (e.g.,  government  agency  or 
branch  of  service)  within  which  many  projects  are  managed  as  a 
whole.  All  projects  within  an  organization  share  a  common  top- 
level  manager  and  common  policies. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

That  part  of  the  appraised  organization  that  constitutes  the  entity 
being  appraised. 

(adapted  from  People  CMM  BAM  Description  [Hefley  98]) 

That  part  of  an  organization  that  is  the  subject  of  an  appraisal  l  also  known 
as  the  organizational  scope  of  the  appraisal).  An  organizational  unit 
deploys  one  or  more  processes  that  have  a  coherent  process  context  and 
operates  within  a  coherent  set  of  business  objectives.  An  organ izat tonal 
unit  is  typically  part  of  a  larger  organization,  although  in  a  small 
oiganization,  the  organizational  unit  might  be  the  whole  organization, 
(derived  from  CMMI  model  glossary  [SEI  02],  ISO  98C  [ISO  98],  and  ARC  VI. 1 
[SEI  01b]) 

“Organizational  unit”  is  not  the  same  as  “unit,”  which  is  a  structural 
subset  of  the  organizational  unit.  (See  “unit.”) 
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A  practice  characterization  value  assigned  to  a  process  instantiation 
when  (1)  direct  artifacts  are  absent  or  judged  to  be  inadequate,  (2) 
artifacts  or  affirmations  suggest  that  some  aspects  of  the  practice 
are  implemented,  and  (3)  weaknesses  have  been  documented. 

(from  MDD  3.7.2  [SEI  01a]) 

Acronym  for  People  CMM  Appraisal  Repository. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  portion  of  the  People  CMM  used  as  a  framework  for  evaluating 
an  organization’s  workforce  practices  during  an  appraisal. 

(from  People  CMM  BAM  Description  (Hefley  98]) 

(See  “partially  implemented.”) 

(See  “practice  implementation  indicator.”) 

The  assignment  of  a  value  describing  the  extent  to  which  a  People 
CMM  model  practice  is  implemented,  used  as  a  mechanism  to 
reach  appraisal  team  consensus.  The  range  of  values  for  practice 
characterization  values  include  fully  implemented  (FI),  largely 
implemented  (LI),  partially  implemented  (PI),  and  not  implemented 
(NI).  Practice  characterization  values  are  assigned  to  each  People 
CMM  model  practice  for  each  process  instantiation  within  the 
appraisal  scope,  and  aggregated  to  the  organizational  unit  level. 

(from  this  document) 

An  objective  attribute  or  characteristic  used  as  a  “footprint”  to 
verify  the  conduct  of  an  activity  or  implementation  of  a  People 
CMM  model  specific  or  generic  practice.  Types  of  practice 
implementation  indicators  include  direct  artifacts,  indirect  artifacts, 
and  affirmations. 

(derived  from  15504-9  [ISO  98]  and  MDD  method  overview  [SEI  01a]) 

Initial  findings  created  by  an  appraisal  team  after  consolidating  and 
synthesizing  valid  observations  to  provide  the  findings  to  appraisal 
participants  for  validation  of  accuracy. 

(derived  from  ARC  VI  .1  [SEI  01  b]) 

In  SCAMPI,  a  source  of  objective  evidence  that  includes 
information  prepared  by  the  organization  and  delivered  visually  or 
verbally  to  the  appraisal  team  to  aid  in  understanding  the 
organizational  processes  and  implementation  of  People  CMM 
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process 
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process 

development 
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model  practices.  This  typically  includes  such  mechanisms  as 
orientation  or  overview  briefings,  and  demonstrations  of  tools  or 
capabilities. 

(derived  from  MDD  method  overview  [SEI  01a]) 

A  cluster  of  related  practices  that,  when  performed  collectively, 
satisfy  a  set  of  goals  that  contribute  to  the  capability  gained  by 
achieving  a  maturity  level. 

(from  The  People  Capability  Maturity  Model:  Guidelines  for  Improving  the  Workforce 
(Curtis  02]) 

A  subset  of  the  appraisal  team  members,  typically  two  or  three,  assigned 
primary  responsibility  for  collection  of  sufficient  appraisal  data  to  ensure 
coverage  of  their  assigned  reference  model  process  areas. 

(from  this  report) 

The  range  of  expected  results  that  can  be  achieved  by  following  a 
process.  (See  “process  performance”  for  contrast.) 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  documented  characterization  of  the  range  of  expected  results  that 
would  normally  be  achieved  by  following  a  specific  process  under 
typical  circumstances.  A  process  capability  baseline  is  typically 
established  at  an  organizational  level. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  operational  definition  of  the  major  components  of  a  process. 
Documentation  that  specifies,  in  a  complete,  precise,  verifiable 
manner,  the  requirements,  design,  behavior,  or  other  characteristics 
of  a  process.  It  might  also  include  the  procedures  for  determining 
whether  these  provisions  have  been  satisfied.  Process  descriptions 
might  be  found  at  the  task,  project,  or  organizational  level. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  act  of  defining  and  describing  a  process.  It  might  include 
planning,  architecture,  design,  implementation,  and  validation. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  extent  to  which  a  specific  process  is  explicitly  defined, 
managed,  measured,  controlled,  and  effective.  Maturity  implies  a 
potential  for  growth  in  software  development  capability.  It  indicates 
both  the  richness  of  an  organization’s  software  process  and  the 
consistency  with  which  the  process  is  applied  on  projects 
throughout  the  organization. 

(from  People  CMM  BAM  Description  [Hefley  98]) 
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The  set  of  definitions,  methods,  and  activities  used  to  take 
measurements  of  a  process  and  its  resulting  products  for  the 
purpose  of  characterizing  and  understanding  the  process. 

(from  People  CMM  BAM  Description  (Hefley  98]) 

A  measure  of  the  actual  results  achieved  by  following  a  process. 
(See  “process  capability”  for  contrast.) 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  set  of  goal  ratings  assigned  to  the  process  areas  in  the  scope  of 
the  appraisal.  In  People  CMM,  also  known  as  the  process  area 
profile. 

(derived  from  ISO  98c  [ISO  98]  and  ARC  VI  .1  [SEI  01b]) 

(See  “appraisal  rating.”) 

Components  of  a  CMM  that  can  be  rated.  These  include  goals,  PAs, 
and  maturity  level. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  rating  scale  for  goals  and  PAs  is  satisfied,  unsatisfied,  not 
applicable,  and  not  rated.  The  rating  scale  for  maturity  level  is  1 
through  5. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Related  to  the  matter  at  hand;  pertinent.  An  observation  must  be 
relevant  to  a  particular  activity,  set  of  activities,  or  common  feature 
to  be  associated  with  a  particular  key  process  area. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  ability  to  attain  appraisal  results  that  accurately  characterize  an 
organization’s  processes. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

The  ability  to  attain  the  same  appraisal  results  if  an  appraisal  of 
identical  scope  is  conducted  more  than  once  in  the  same  time 
period. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Rules  that  define  the  requirements  for  confirming  observations 
through  the  use  of  multiple  data  sources  and  sessions  of  prescribed 
types. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  set  of  elements  drawn  from  and  analyzed  to  estimate  the 
characteristics  of  a  population.  During  an  appraisal,  data  collection 
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is  planned  to  provide  a  sampling  of  the  process  data  related  to  the 
People  CMM  key  process  areas,  organizational  units,  and 
individuals  and  managers  within  the  scope  of  the  appraisal. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Rating  given  to  a  goal  when  the  aggregate  of  valid  observations  and 
associated  findings  does  not  negatively  impact  achievement  of  the 
goal.  Rating  given  to  a  process  area  when  all  of  its  goals  are  rated 
“satisfied.” 

(from  ARC  VI. 1  [SEI  01b]) 

(See  “appraisal  scope.”) 

Acronym  for  Software  Engineering  Institute. 

Senior  manager,  in  charge  of  the  appraised  entity  and  usually  the 
sponsor  of  the  appraisal,  who  identifies  the  business  goals  that  bear 
on  the  organization’s  software  development  and  maintenance 
activity,  identifies  the  scope  of  the  appraisal  and  any  constraints  that 
will  exist,  gives  the  team  leader  the  authorization  to  proceed,  and 
personally  participates  in  the  opening  meeting  and  final  findings 
presentation. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Having  or  expressing  a  meaning;  meaningful;  notable;  valuable.  An 
observation  is  significant  if  it  is  evidence  of  a  strength  or  a 
weakness  of  a  People  CMM  component.  If  an  observation  is  not  a 
strength  or  weakness,  it  could  be  significant  as  an  acceptable 
alternative  to  the  defined  People  CMM  practice. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  geographic  location  of  one  or  more  of  an  organization’s  units. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  member  of  the  appraisal  team  who  is  responsible  for  handling  the 
logistics  of  the  appraisal,  including  developing  the  schedule, 
notifying  appraisal  participants,  reserving  rooms,  scheduling 
contingency  interviews,  and  handling  supplies  and  meals. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

Information  to  help  ensure  that  appraisal  team  members  are 
prepared  to  gather,  interpret,  and  understand  the  information  they 
receive  during  the  course  of  an  appraisal.  The  package  assists  the 
team  members  who  might  not  be  from  the  site  in  understanding  the 
organization’s  culture,  the  language  of  the  site’s  software 
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practitioners,  and  the  context  within  which  they  operate. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

(See  “appraisal  sponsor”  and  “senior  site  manager.”) 

Exemplary  or  noteworthy  implementation  of  a  People  CMM  model 
practice. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI. 1  [SEI  01b]) 

Description  of  what  one  would  expect  to  find  implemented  for  the 
top-level  key  practice.  The  subpractices  are  listed  beneath  top  level 
key  practices  in  the  People  CMM  and  can  be  used  to  assist  in 
determining  whether  or  not  the  key  practices  are  implemented 
satisfactorily.  Subpractices  are  not  rating  components  of  the  People 
CMM. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  determination  that  the  coverage  requirements  have  been  met.  (See 
“coverage”  and  “coverage  criteria.”) 

(from  ARC  VI. 1  [SEI  01b]) 

(See  “appraisal  tailoring.”) 

The  degree  to  which  a  relationship  can  be  established  between  two 
or  more  products  of  the  appraisal  process,  especially  products 
having  a  predecessor-successor  relationship  to  one  another. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

A  single,  well-defined  organizational  component  (e.g..  a 
department,  section,  or  project)  of  an  organization.  The  term  “unit” 
is  used  to  refer  to  any  organizational  entity  that  is  accountable  to  a 
specified  individual(s)  (usually  a  manager)  responsible  for 
accomplishing  a  set  of  performance  objectives  that  can  be  met  only 
through  collective  action.  A  workgroup  may  constitute  the  lowest- 
level  unit,  but  the  lowest-level  units  often  consist  of  several 
workgroups.  “Unit”  is  a  recursive  concept;  units  may  be  composed 
from  other  units  cascading  down  the  organization  For  instance,  a 
division  may  be  a  unit  consisting  of  departments,  each  of  which 
may  be  a  unit  consisting  of  programs,  each  of  which  may  be  a  unit 
consisting  of  projects,  and  so  on. 

(from  The  People  Capability  Maturity  Model:  Guidelines  for  Improving  the  Workforce 
[Curtis  02]) 

Rating  given  to  a  People  CMM  component  that  is  both  applicable 
and  significant  in  an  organization’s  business  environment,  is  either 
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valid  observation 


verification-based 

appraisal 


weakness 


not  performed  or  is  performed  as  defined  in  the  People  CMM  with 
significant  weaknesses,  and  for  which  no  adequate  alternative 
exists. 

(from  People  CMM  BAM  Description  [Hefley  98]) 

An  observation  that  the  appraisal  team  members  agree  is  (a) 
accurate,  (b)  corroborated,  and  (c)  consistent  with  other  valid 
observations. 

(from  ARC  VI. 1  [SEIOlb]) 

An  appraisal  in  which  the  focus  of  the  appraisal  team  is  on 
verifying  the  set  of  objective  evidence  provided  by  the  appraised 
organization  in  advance  of  the  appraisal  in  order  to  reduce  the 
amount  of  probing  and  discovery  of  objective  evidence  during  the 
appraisal  on-site  period.  (See  “discovery-based  appraisal”  for 
contrast.) 

(from  MDD  method  overview  [SEI  01a]) 

The  ineffective,  or  lack  of,  implementation  of  one  or  more  People 
CMM  model  practices. 

(from  CMMI  model  glossary  [SEI  02]  and  ARC  VI  . 1  [SEI  01b]) 
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